[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782018616648578-3598-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 21 05:10:16 dist-test-slave-vrq1 krb5kdc[3605](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 21 05:10:16 dist-test-slave-vrq1 krb5kdc[3605](info): set up 2 sockets
Jun 21 05:10:16 dist-test-slave-vrq1 krb5kdc[3605](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260621 05:10:17.751999 3598 mini_kdc.cc:121] Time spent starting KDC: real 1.082s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782018616648578-3598-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782018616648578-3598-0/krb5kdc/test-user.keytab.
Jun 21 05:10:17 dist-test-slave-vrq1 krb5kdc[3605](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782018617, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-21T05:10:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-21T05:10:17Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.131.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.131.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.131.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782018616648578-3598-0/krb5kdc/kudu_127.3.131.190.keytab.
Entry for principal kudu/127.3.131.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782018616648578-3598-0/krb5kdc/kudu_127.3.131.190.keytab.
I20260621 05:10:17.879148 3598 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWZOx2_/build/release/bin/kudu
/tmp/dist-test-taskWZOx2_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782018616648578-3598-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782018616648578-3598-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782018616648578-3598-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782018616648578-3598-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.131.190:41035
--webserver_interface=127.3.131.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782018616648578-3598-0/krb5kdc/kudu_127.3.131.190.keytab
--principal=kudu/127.3.131.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.131.148:40287
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.131.190:41035
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782018616648578-3598-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782018616648578-3598-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782018616648578-3598-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 05:10:17.968727 3618 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 05:10:17.969041 3618 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 05:10:17.969086 3618 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 05:10:17.971323 3618 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 05:10:17.971432 3618 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 05:10:17.971459 3618 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 05:10:17.971486 3618 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 05:10:17.971519 3618 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 05:10:17.973564 3618 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.131.148:40287
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782018616648578-3598-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782018616648578-3598-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.131.190:41035
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782018616648578-3598-0/krb5kdc/kudu_127.3.131.190.keytab
--principal=kudu/127.3.131.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.131.190:41035
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782018616648578-3598-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.131.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3618
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782018616648578-3598-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 21 Jun 2026 04:16:16 UTC on ccycloud.quasar-xycaha.root.comops.site
build id 4457881
I20260621 05:10:17.973941 3618 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 05:10:17.974247 3618 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260621 05:10:17.981850 3618 server_base.cc:1034] running on GCE node
W20260621 05:10:17.981679 3624 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:10:17.981679 3623 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:10:17.981784 3626 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:10:17.982242 3618 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 05:10:17.982628 3618 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 05:10:17.983842 3618 hybrid_clock.cc:648] HybridClock initialized: now 1782018617983812 us; error 39 us; skew 500 ppm
Jun 21 05:10:17 dist-test-slave-vrq1 krb5kdc[3605](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782018617, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.131.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 05:10:17.986755 3618 init.cc:377] Logged in from keytab as kudu/127.3.131.190@KRBTEST.COM (short username kudu)
I20260621 05:10:17.987365 3618 webserver.cc:460] Webserver started at http://127.3.131.190:37451/ using document root <none> and password file <none>
I20260621 05:10:17.987641 3618 fs_manager.cc:359] Metadata directory not provided
I20260621 05:10:17.987715 3618 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 05:10:17.987871 3618 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 05:10:17.989272 3618 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782018616648578-3598-0/minicluster-data/master-0/data/instance:
uuid: "01cc8ea392754dd5a3f11a600bcbcffc"
format_stamp: "Formatted at 2026-06-21 05:10:17 on dist-test-slave-vrq1"
server_key: "dfebe0a94342bc9e48a575163d69ea01"
server_key_iv: "2639c0a5b93b29feaaca3ae161daa00f"
server_key_version: "encryptionkey@0"
I20260621 05:10:17.989949 3618 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782018616648578-3598-0/minicluster-data/master-0/wal/instance:
uuid: "01cc8ea392754dd5a3f11a600bcbcffc"
format_stamp: "Formatted at 2026-06-21 05:10:17 on dist-test-slave-vrq1"
server_key: "dfebe0a94342bc9e48a575163d69ea01"
server_key_iv: "2639c0a5b93b29feaaca3ae161daa00f"
server_key_version: "encryptionkey@0"
I20260621 05:10:17.991981 3618 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.001s sys 0.003s
I20260621 05:10:17.992939 3633 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 05:10:17.993198 3618 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260621 05:10:17.993326 3618 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782018616648578-3598-0/minicluster-data/master-0/data,/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782018616648578-3598-0/minicluster-data/master-0/wal
uuid: "01cc8ea392754dd5a3f11a600bcbcffc"
format_stamp: "Formatted at 2026-06-21 05:10:17 on dist-test-slave-vrq1"
server_key: "dfebe0a94342bc9e48a575163d69ea01"
server_key_iv: "2639c0a5b93b29feaaca3ae161daa00f"
server_key_version: "encryptionkey@0"
I20260621 05:10:17.993409 3618 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782018616648578-3598-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782018616648578-3598-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782018616648578-3598-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 05:10:18.006909 3618 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 05:10:18.009845 3618 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 05:10:18.010042 3618 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 05:10:18.014580 3618 rpc_server.cc:307] RPC server started. Bound to: 127.3.131.190:41035
I20260621 05:10:18.014614 3685 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.131.190:41035 every 8 connection(s)
I20260621 05:10:18.015033 3618 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782018616648578-3598-0/minicluster-data/master-0/data/info.pb
I20260621 05:10:18.015967 3686 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 05:10:18.018603 3686 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 01cc8ea392754dd5a3f11a600bcbcffc: Bootstrap starting.
I20260621 05:10:18.019268 3686 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 01cc8ea392754dd5a3f11a600bcbcffc: Neither blocks nor log segments found. Creating new log.
I20260621 05:10:18.019533 3686 log.cc:824] T 00000000000000000000000000000000 P 01cc8ea392754dd5a3f11a600bcbcffc: Log is configured to *not* fsync() on all Append() calls
I20260621 05:10:18.020319 3686 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 01cc8ea392754dd5a3f11a600bcbcffc: No bootstrap required, opened a new log
I20260621 05:10:18.021943 3686 raft_consensus.cc:348] T 00000000000000000000000000000000 P 01cc8ea392754dd5a3f11a600bcbcffc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "01cc8ea392754dd5a3f11a600bcbcffc" member_type: VOTER last_known_addr { host: "127.3.131.190" port: 41035 } }
I20260621 05:10:18.022097 3686 raft_consensus.cc:374] T 00000000000000000000000000000000 P 01cc8ea392754dd5a3f11a600bcbcffc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 05:10:18.022122 3686 raft_consensus.cc:729] T 00000000000000000000000000000000 P 01cc8ea392754dd5a3f11a600bcbcffc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 01cc8ea392754dd5a3f11a600bcbcffc, State: Initialized, Role: FOLLOWER
I20260621 05:10:18.022277 3686 consensus_queue.cc:260] T 00000000000000000000000000000000 P 01cc8ea392754dd5a3f11a600bcbcffc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "01cc8ea392754dd5a3f11a600bcbcffc" member_type: VOTER last_known_addr { host: "127.3.131.190" port: 41035 } }
I20260621 05:10:18.022363 3686 raft_consensus.cc:388] T 00000000000000000000000000000000 P 01cc8ea392754dd5a3f11a600bcbcffc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260621 05:10:18.022387 3686 raft_consensus.cc:482] T 00000000000000000000000000000000 P 01cc8ea392754dd5a3f11a600bcbcffc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260621 05:10:18.022416 3686 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 01cc8ea392754dd5a3f11a600bcbcffc [term 0 FOLLOWER]: Advancing to term 1
I20260621 05:10:18.022935 3686 raft_consensus.cc:504] T 00000000000000000000000000000000 P 01cc8ea392754dd5a3f11a600bcbcffc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "01cc8ea392754dd5a3f11a600bcbcffc" member_type: VOTER last_known_addr { host: "127.3.131.190" port: 41035 } }
I20260621 05:10:18.023022 3686 leader_election.cc:302] T 00000000000000000000000000000000 P 01cc8ea392754dd5a3f11a600bcbcffc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 01cc8ea392754dd5a3f11a600bcbcffc; no voters:
I20260621 05:10:18.023161 3686 leader_election.cc:288] T 00000000000000000000000000000000 P 01cc8ea392754dd5a3f11a600bcbcffc [CANDIDATE]: Term 1 election: Requested vote from peers
I20260621 05:10:18.023272 3689 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 01cc8ea392754dd5a3f11a600bcbcffc [term 1 FOLLOWER]: Leader election won for term 1
I20260621 05:10:18.023464 3689 raft_consensus.cc:686] T 00000000000000000000000000000000 P 01cc8ea392754dd5a3f11a600bcbcffc [term 1 LEADER]: Becoming Leader. State: Replica: 01cc8ea392754dd5a3f11a600bcbcffc, State: Running, Role: LEADER
I20260621 05:10:18.023505 3686 sys_catalog.cc:564] T 00000000000000000000000000000000 P 01cc8ea392754dd5a3f11a600bcbcffc [sys.catalog]: configured and running, proceeding with master startup.
I20260621 05:10:18.023573 3689 consensus_queue.cc:237] T 00000000000000000000000000000000 P 01cc8ea392754dd5a3f11a600bcbcffc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "01cc8ea392754dd5a3f11a600bcbcffc" member_type: VOTER last_known_addr { host: "127.3.131.190" port: 41035 } }
I20260621 05:10:18.024084 3691 sys_catalog.cc:455] T 00000000000000000000000000000000 P 01cc8ea392754dd5a3f11a600bcbcffc [sys.catalog]: SysCatalogTable state changed. Reason: New leader 01cc8ea392754dd5a3f11a600bcbcffc. Latest consensus state: current_term: 1 leader_uuid: "01cc8ea392754dd5a3f11a600bcbcffc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "01cc8ea392754dd5a3f11a600bcbcffc" member_type: VOTER last_known_addr { host: "127.3.131.190" port: 41035 } } }
I20260621 05:10:18.024191 3691 sys_catalog.cc:458] T 00000000000000000000000000000000 P 01cc8ea392754dd5a3f11a600bcbcffc [sys.catalog]: This master's current role is: LEADER
I20260621 05:10:18.024230 3690 sys_catalog.cc:455] T 00000000000000000000000000000000 P 01cc8ea392754dd5a3f11a600bcbcffc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "01cc8ea392754dd5a3f11a600bcbcffc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "01cc8ea392754dd5a3f11a600bcbcffc" member_type: VOTER last_known_addr { host: "127.3.131.190" port: 41035 } } }
I20260621 05:10:18.024325 3690 sys_catalog.cc:458] T 00000000000000000000000000000000 P 01cc8ea392754dd5a3f11a600bcbcffc [sys.catalog]: This master's current role is: LEADER
I20260621 05:10:18.024456 3698 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260621 05:10:18.025024 3698 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260621 05:10:18.025007 3598 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWZOx2_/build/release/bin/kudu as pid 3618
I20260621 05:10:18.025154 3598 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782018616648578-3598-0/minicluster-data/master-0/wal/instance
I20260621 05:10:18.025902 3598 external_mini_cluster.cc:1428] Setting key f5c1ca83696896b4628f5f3c1743c02b
I20260621 05:10:18.027128 3698 catalog_manager.cc:1347] Generated new cluster ID: 3b08c8a6ff32481798ab192ab4ff9fe5
I20260621 05:10:18.027208 3698 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Jun 21 05:10:18 dist-test-slave-vrq1 krb5kdc[3605](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782018617, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.131.190@KRBTEST.COM
I20260621 05:10:18.041640 3698 catalog_manager.cc:1370] Generated new certificate authority record
I20260621 05:10:18.042354 3698 catalog_manager.cc:1504] Loading token signing keys...
I20260621 05:10:18.047390 3708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:10:18.027026 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43674 (local address 127.3.131.190:41035)
0621 05:10:18.027519 (+ 493us) server_negotiation.cc:206] Beginning negotiation
0621 05:10:18.027528 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0621 05:10:18.027574 (+ 46us) server_negotiation.cc:407] Connection header received
0621 05:10:18.029937 (+ 2363us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:10:18.029945 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:10:18.030020 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:10:18.030179 (+ 159us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:10:18.032584 (+ 2405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:10:18.034254 (+ 1670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:10:18.036122 (+ 1868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:10:18.036809 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:10:18.036926 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:10:18.041796 (+ 4870us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:10:18.041801 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:10:18.041802 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:10:18.041817 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:10:18.044406 (+ 2589us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:10:18.045025 (+ 619us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:10:18.045029 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:10:18.045030 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:10:18.045181 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:10:18.045542 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:10:18.045543 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:10:18.045543 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:10:18.045884 (+ 341us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:10:18.046066 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0621 05:10:18.046504 (+ 438us) server_negotiation.cc:299] Negotiation successful
0621 05:10:18.047083 (+ 579us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":443,"thread_start_us":72,"threads_started":1}
I20260621 05:10:18.048887 3698 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 01cc8ea392754dd5a3f11a600bcbcffc: Generated new TSK 0
I20260621 05:10:18.049098 3698 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.3.131.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.131.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.131.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782018616648578-3598-0/krb5kdc/kudu_127.3.131.129.keytab.
Entry for principal kudu/127.3.131.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782018616648578-3598-0/krb5kdc/kudu_127.3.131.129.keytab.
I20260621 05:10:18.086771 3598 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWZOx2_/build/release/bin/kudu
/tmp/dist-test-taskWZOx2_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782018616648578-3598-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782018616648578-3598-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782018616648578-3598-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782018616648578-3598-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.131.129:0
--local_ip_for_outbound_sockets=127.3.131.129
--webserver_interface=127.3.131.129
--webserver_port=0
--tserver_master_addrs=127.3.131.190:41035
--keytab_file=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782018616648578-3598-0/krb5kdc/kudu_127.3.131.129.keytab
--principal=kudu/127.3.131.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.131.148:40287
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782018616648578-3598-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782018616648578-3598-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782018616648578-3598-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 05:10:18.193567 3712 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 05:10:18.193792 3712 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 05:10:18.193818 3712 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 05:10:18.194945 3712 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 05:10:18.194985 3712 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 05:10:18.195012 3712 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.131.129
I20260621 05:10:18.196626 3712 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.131.148:40287
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782018616648578-3598-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782018616648578-3598-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782018616648578-3598-0/krb5kdc/kudu_127.3.131.129.keytab
--principal=kudu/127.3.131.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.131.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782018616648578-3598-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.131.129
--webserver_port=0
--tserver_master_addrs=127.3.131.190:41035
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3712
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.131.129
--log_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782018616648578-3598-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 21 Jun 2026 04:16:16 UTC on ccycloud.quasar-xycaha.root.comops.site
build id 4457881
I20260621 05:10:18.196900 3712 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 05:10:18.197103 3712 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 05:10:18.202536 3718 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:10:18.202562 3717 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:10:18.202518 3720 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:10:18.202737 3712 server_base.cc:1034] running on GCE node
I20260621 05:10:18.203040 3712 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 05:10:18.203325 3712 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 05:10:18.204504 3712 hybrid_clock.cc:648] HybridClock initialized: now 1782018618204478 us; error 40 us; skew 500 ppm
Jun 21 05:10:18 dist-test-slave-vrq1 krb5kdc[3605](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782018618, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.131.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 05:10:18.207634 3712 init.cc:377] Logged in from keytab as kudu/127.3.131.129@KRBTEST.COM (short username kudu)
I20260621 05:10:18.208230 3712 webserver.cc:460] Webserver started at http://127.3.131.129:42639/ using document root <none> and password file <none>
I20260621 05:10:18.208489 3712 fs_manager.cc:359] Metadata directory not provided
I20260621 05:10:18.208537 3712 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 05:10:18.208658 3712 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 05:10:18.209825 3712 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782018616648578-3598-0/minicluster-data/ts-0/data/instance:
uuid: "569135a5864046f8a3eee85b791586d6"
format_stamp: "Formatted at 2026-06-21 05:10:18 on dist-test-slave-vrq1"
server_key: "95dc07fc5fc8c00bc9e0e647e6e40235"
server_key_iv: "b1252e0e9845af221b0b1f6bd5b6a67f"
server_key_version: "encryptionkey@0"
I20260621 05:10:18.210199 3712 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782018616648578-3598-0/minicluster-data/ts-0/wal/instance:
uuid: "569135a5864046f8a3eee85b791586d6"
format_stamp: "Formatted at 2026-06-21 05:10:18 on dist-test-slave-vrq1"
server_key: "95dc07fc5fc8c00bc9e0e647e6e40235"
server_key_iv: "b1252e0e9845af221b0b1f6bd5b6a67f"
server_key_version: "encryptionkey@0"
I20260621 05:10:18.211660 3712 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260621 05:10:18.212541 3727 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 05:10:18.212730 3712 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260621 05:10:18.212837 3712 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782018616648578-3598-0/minicluster-data/ts-0/data,/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782018616648578-3598-0/minicluster-data/ts-0/wal
uuid: "569135a5864046f8a3eee85b791586d6"
format_stamp: "Formatted at 2026-06-21 05:10:18 on dist-test-slave-vrq1"
server_key: "95dc07fc5fc8c00bc9e0e647e6e40235"
server_key_iv: "b1252e0e9845af221b0b1f6bd5b6a67f"
server_key_version: "encryptionkey@0"
I20260621 05:10:18.212925 3712 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782018616648578-3598-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782018616648578-3598-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782018616648578-3598-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 05:10:18.232685 3712 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 05:10:18.235320 3712 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 05:10:18.235507 3712 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 05:10:18.235800 3712 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 05:10:18.236167 3712 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 05:10:18.236231 3712 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 05:10:18.236274 3712 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 05:10:18.236404 3712 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 05:10:18.242383 3712 rpc_server.cc:307] RPC server started. Bound to: 127.3.131.129:35613
I20260621 05:10:18.242470 3840 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.131.129:35613 every 8 connection(s)
I20260621 05:10:18.242817 3712 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782018616648578-3598-0/minicluster-data/ts-0/data/info.pb
I20260621 05:10:18.243285 3598 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWZOx2_/build/release/bin/kudu as pid 3712
I20260621 05:10:18.243381 3598 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782018616648578-3598-0/minicluster-data/ts-0/wal/instance
I20260621 05:10:18.243539 3598 external_mini_cluster.cc:1428] Setting key bff62dd675e2ea21e3cacc6dccce281f
Jun 21 05:10:18 dist-test-slave-vrq1 krb5kdc[3605](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782018618, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.131.129@KRBTEST.COM for kudu/127.3.131.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.3.131.130@KRBTEST.COM; defaulting to no policy
I20260621 05:10:18.255064 3708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:10:18.243855 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.131.129:39351 (local address 127.3.131.190:41035)
0621 05:10:18.243909 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0621 05:10:18.243912 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0621 05:10:18.244157 (+ 245us) server_negotiation.cc:407] Connection header received
0621 05:10:18.244733 (+ 576us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:10:18.244734 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:10:18.244746 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:10:18.244816 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:10:18.246137 (+ 1321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:10:18.246639 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:10:18.248282 (+ 1643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:10:18.248500 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:10:18.248566 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:10:18.251471 (+ 2905us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:10:18.251473 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:10:18.251473 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:10:18.251483 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:10:18.253153 (+ 1670us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:10:18.253761 (+ 608us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:10:18.253763 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:10:18.253763 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:10:18.253830 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:10:18.254336 (+ 506us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:10:18.254337 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:10:18.254337 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:10:18.254462 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:10:18.254521 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0621 05:10:18.254917 (+ 396us) server_negotiation.cc:299] Negotiation successful
0621 05:10:18.254996 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.131.130@KRBTEST.COM" created.
I20260621 05:10:18.255453 3843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:10:18.243836 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.131.190:41035 (local address 127.3.131.129:39351)
0621 05:10:18.244010 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0621 05:10:18.244057 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0621 05:10:18.244627 (+ 570us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:10:18.244873 (+ 246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:10:18.244876 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:10:18.245719 (+ 843us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:10:18.246023 (+ 304us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:10:18.246030 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:10:18.246715 (+ 685us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:10:18.246716 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:10:18.248158 (+ 1442us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:10:18.248162 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:10:18.248598 (+ 436us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:10:18.248599 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:10:18.248762 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:10:18.249391 (+ 629us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 05:10:18.249403 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:10:18.251352 (+ 1949us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 05:10:18.253272 (+ 1920us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:10:18.253280 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:10:18.253283 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:10:18.253618 (+ 335us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:10:18.254066 (+ 448us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:10:18.254067 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:10:18.254067 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:10:18.254233 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:10:18.254591 (+ 358us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 05:10:18.254599 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 05:10:18.254786 (+ 187us) client_negotiation.cc:769] Sending connection context
0621 05:10:18.254888 (+ 102us) client_negotiation.cc:240] Negotiation successful
0621 05:10:18.255133 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":90,"threads_started":1}
I20260621 05:10:18.256055 3841 heartbeater.cc:347] Connected to a master server at 127.3.131.190:41035
I20260621 05:10:18.256172 3841 heartbeater.cc:464] Registering TS with master...
I20260621 05:10:18.256385 3841 heartbeater.cc:511] Master 127.3.131.190:41035 requested a full tablet report, sending...
I20260621 05:10:18.257051 3650 ts_manager.cc:194] Registered new tserver with Master: 569135a5864046f8a3eee85b791586d6 (127.3.131.129:35613)
I20260621 05:10:18.258070 3650 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.131.129@KRBTEST.COM'} at 127.3.131.129:39351
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.131.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782018616648578-3598-0/krb5kdc/kudu_127.3.131.130.keytab.
Entry for principal kudu/127.3.131.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782018616648578-3598-0/krb5kdc/kudu_127.3.131.130.keytab.
I20260621 05:10:18.273914 3598 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWZOx2_/build/release/bin/kudu
/tmp/dist-test-taskWZOx2_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782018616648578-3598-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782018616648578-3598-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782018616648578-3598-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782018616648578-3598-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.131.130:0
--local_ip_for_outbound_sockets=127.3.131.130
--webserver_interface=127.3.131.130
--webserver_port=0
--tserver_master_addrs=127.3.131.190:41035
--keytab_file=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782018616648578-3598-0/krb5kdc/kudu_127.3.131.130.keytab
--principal=kudu/127.3.131.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.131.148:40287
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782018616648578-3598-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782018616648578-3598-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782018616648578-3598-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 05:10:18.356734 3846 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 05:10:18.356959 3846 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 05:10:18.356985 3846 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 05:10:18.358109 3846 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 05:10:18.358145 3846 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 05:10:18.358172 3846 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.131.130
I20260621 05:10:18.359717 3846 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.131.148:40287
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782018616648578-3598-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782018616648578-3598-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782018616648578-3598-0/krb5kdc/kudu_127.3.131.130.keytab
--principal=kudu/127.3.131.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.131.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782018616648578-3598-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.131.130
--webserver_port=0
--tserver_master_addrs=127.3.131.190:41035
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3846
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.131.130
--log_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782018616648578-3598-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 21 Jun 2026 04:16:16 UTC on ccycloud.quasar-xycaha.root.comops.site
build id 4457881
I20260621 05:10:18.359935 3846 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 05:10:18.360141 3846 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 05:10:18.366591 3852 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:10:18.366652 3851 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:10:18.366829 3854 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:10:18.367587 3846 server_base.cc:1034] running on GCE node
I20260621 05:10:18.367978 3846 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 05:10:18.368346 3846 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 05:10:18.369547 3846 hybrid_clock.cc:648] HybridClock initialized: now 1782018618369518 us; error 96 us; skew 500 ppm
Jun 21 05:10:18 dist-test-slave-vrq1 krb5kdc[3605](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782018618, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.131.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 05:10:18.372802 3846 init.cc:377] Logged in from keytab as kudu/127.3.131.130@KRBTEST.COM (short username kudu)
I20260621 05:10:18.373284 3846 webserver.cc:460] Webserver started at http://127.3.131.130:39755/ using document root <none> and password file <none>
I20260621 05:10:18.373498 3846 fs_manager.cc:359] Metadata directory not provided
I20260621 05:10:18.373543 3846 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 05:10:18.373643 3846 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 05:10:18.374753 3846 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782018616648578-3598-0/minicluster-data/ts-1/data/instance:
uuid: "1dc8ab3deecd4af9bd132b6c588e0c47"
format_stamp: "Formatted at 2026-06-21 05:10:18 on dist-test-slave-vrq1"
server_key: "a09b23cf8a9e3dcd8b35a10ff94890ba"
server_key_iv: "ad27324c9bdbfef5b71b8a9fe7966fe1"
server_key_version: "encryptionkey@0"
I20260621 05:10:18.375087 3846 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782018616648578-3598-0/minicluster-data/ts-1/wal/instance:
uuid: "1dc8ab3deecd4af9bd132b6c588e0c47"
format_stamp: "Formatted at 2026-06-21 05:10:18 on dist-test-slave-vrq1"
server_key: "a09b23cf8a9e3dcd8b35a10ff94890ba"
server_key_iv: "ad27324c9bdbfef5b71b8a9fe7966fe1"
server_key_version: "encryptionkey@0"
I20260621 05:10:18.376641 3846 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260621 05:10:18.377516 3861 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 05:10:18.377770 3846 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260621 05:10:18.377872 3846 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782018616648578-3598-0/minicluster-data/ts-1/data,/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782018616648578-3598-0/minicluster-data/ts-1/wal
uuid: "1dc8ab3deecd4af9bd132b6c588e0c47"
format_stamp: "Formatted at 2026-06-21 05:10:18 on dist-test-slave-vrq1"
server_key: "a09b23cf8a9e3dcd8b35a10ff94890ba"
server_key_iv: "ad27324c9bdbfef5b71b8a9fe7966fe1"
server_key_version: "encryptionkey@0"
I20260621 05:10:18.377941 3846 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782018616648578-3598-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782018616648578-3598-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782018616648578-3598-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 05:10:18.394671 3846 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 05:10:18.397495 3846 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 05:10:18.397655 3846 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 05:10:18.397856 3846 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 05:10:18.398212 3846 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 05:10:18.398276 3846 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 05:10:18.398322 3846 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 05:10:18.398355 3846 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 05:10:18.406694 3846 rpc_server.cc:307] RPC server started. Bound to: 127.3.131.130:34191
I20260621 05:10:18.407166 3846 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782018616648578-3598-0/minicluster-data/ts-1/data/info.pb
I20260621 05:10:18.407552 3974 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.131.130:34191 every 8 connection(s)
I20260621 05:10:18.409474 3598 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWZOx2_/build/release/bin/kudu as pid 3846
I20260621 05:10:18.409595 3598 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782018616648578-3598-0/minicluster-data/ts-1/wal/instance
I20260621 05:10:18.409775 3598 external_mini_cluster.cc:1428] Setting key 8ab109e5a0b417e7a11f8b25d362ba90
Jun 21 05:10:18 dist-test-slave-vrq1 krb5kdc[3605](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782018618, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.131.130@KRBTEST.COM for kudu/127.3.131.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.3.131.131@KRBTEST.COM; defaulting to no policy
I20260621 05:10:18.419550 3708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:10:18.408357 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.131.130:54037 (local address 127.3.131.190:41035)
0621 05:10:18.408452 (+ 95us) server_negotiation.cc:206] Beginning negotiation
0621 05:10:18.408455 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0621 05:10:18.409530 (+ 1075us) server_negotiation.cc:407] Connection header received
0621 05:10:18.410209 (+ 679us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:10:18.410210 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:10:18.410219 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:10:18.410283 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:10:18.411427 (+ 1144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:10:18.412154 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:10:18.413224 (+ 1070us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:10:18.413436 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:10:18.413515 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:10:18.416237 (+ 2722us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:10:18.416238 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:10:18.416239 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:10:18.416247 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:10:18.417925 (+ 1678us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:10:18.418450 (+ 525us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:10:18.418451 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:10:18.418452 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:10:18.418518 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:10:18.418795 (+ 277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:10:18.418796 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:10:18.418797 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:10:18.418926 (+ 129us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:10:18.419006 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0621 05:10:18.419360 (+ 354us) server_negotiation.cc:299] Negotiation successful
0621 05:10:18.419451 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260621 05:10:18.419772 3977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:10:18.408337 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.131.190:41035 (local address 127.3.131.130:54037)
0621 05:10:18.409373 (+ 1036us) negotiation.cc:107] Waiting for socket to connect
0621 05:10:18.409427 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0621 05:10:18.410108 (+ 681us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:10:18.410368 (+ 260us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:10:18.410373 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:10:18.411028 (+ 655us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:10:18.411321 (+ 293us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:10:18.411332 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:10:18.412265 (+ 933us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:10:18.412266 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:10:18.413110 (+ 844us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:10:18.413112 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:10:18.413535 (+ 423us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:10:18.413536 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:10:18.413683 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:10:18.414300 (+ 617us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 05:10:18.414309 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:10:18.416092 (+ 1783us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 05:10:18.418026 (+ 1934us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:10:18.418036 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:10:18.418038 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:10:18.418368 (+ 330us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:10:18.418589 (+ 221us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:10:18.418590 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:10:18.418590 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:10:18.418724 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:10:18.418996 (+ 272us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 05:10:18.419003 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 05:10:18.419211 (+ 208us) client_negotiation.cc:769] Sending connection context
0621 05:10:18.419337 (+ 126us) client_negotiation.cc:240] Negotiation successful
0621 05:10:18.419513 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":990,"thread_start_us":82,"threads_started":1}
I20260621 05:10:18.420349 3975 heartbeater.cc:347] Connected to a master server at 127.3.131.190:41035
I20260621 05:10:18.420459 3975 heartbeater.cc:464] Registering TS with master...
I20260621 05:10:18.420675 3975 heartbeater.cc:511] Master 127.3.131.190:41035 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.131.131@KRBTEST.COM" created.
I20260621 05:10:18.421108 3650 ts_manager.cc:194] Registered new tserver with Master: 1dc8ab3deecd4af9bd132b6c588e0c47 (127.3.131.130:34191)
I20260621 05:10:18.421531 3650 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.131.130@KRBTEST.COM'} at 127.3.131.130:54037
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.131.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782018616648578-3598-0/krb5kdc/kudu_127.3.131.131.keytab.
Entry for principal kudu/127.3.131.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782018616648578-3598-0/krb5kdc/kudu_127.3.131.131.keytab.
I20260621 05:10:18.443801 3598 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWZOx2_/build/release/bin/kudu
/tmp/dist-test-taskWZOx2_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782018616648578-3598-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782018616648578-3598-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782018616648578-3598-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782018616648578-3598-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.131.131:0
--local_ip_for_outbound_sockets=127.3.131.131
--webserver_interface=127.3.131.131
--webserver_port=0
--tserver_master_addrs=127.3.131.190:41035
--keytab_file=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782018616648578-3598-0/krb5kdc/kudu_127.3.131.131.keytab
--principal=kudu/127.3.131.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.131.148:40287
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782018616648578-3598-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782018616648578-3598-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782018616648578-3598-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 05:10:18.531729 3980 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 05:10:18.531963 3980 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 05:10:18.531989 3980 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 05:10:18.533181 3980 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 05:10:18.533221 3980 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 05:10:18.533249 3980 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.131.131
I20260621 05:10:18.534791 3980 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.131.148:40287
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782018616648578-3598-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782018616648578-3598-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782018616648578-3598-0/krb5kdc/kudu_127.3.131.131.keytab
--principal=kudu/127.3.131.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.131.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782018616648578-3598-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.131.131
--webserver_port=0
--tserver_master_addrs=127.3.131.190:41035
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3980
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.131.131
--log_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782018616648578-3598-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 21 Jun 2026 04:16:16 UTC on ccycloud.quasar-xycaha.root.comops.site
build id 4457881
I20260621 05:10:18.535010 3980 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 05:10:18.535296 3980 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 05:10:18.540794 3986 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:10:18.540868 3980 server_base.cc:1034] running on GCE node
W20260621 05:10:18.540789 3985 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:10:18.540794 3988 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:10:18.541399 3980 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 05:10:18.541630 3980 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 05:10:18.542810 3980 hybrid_clock.cc:648] HybridClock initialized: now 1782018618542796 us; error 27 us; skew 500 ppm
Jun 21 05:10:18 dist-test-slave-vrq1 krb5kdc[3605](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782018618, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.131.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 05:10:18.545670 3980 init.cc:377] Logged in from keytab as kudu/127.3.131.131@KRBTEST.COM (short username kudu)
I20260621 05:10:18.546190 3980 webserver.cc:460] Webserver started at http://127.3.131.131:39135/ using document root <none> and password file <none>
I20260621 05:10:18.546432 3980 fs_manager.cc:359] Metadata directory not provided
I20260621 05:10:18.546478 3980 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 05:10:18.546612 3980 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 05:10:18.547822 3980 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782018616648578-3598-0/minicluster-data/ts-2/data/instance:
uuid: "0e0c3767fccc47b4ae51d70fb78bfb23"
format_stamp: "Formatted at 2026-06-21 05:10:18 on dist-test-slave-vrq1"
server_key: "a039e6e989e2a8733524568be3cf1d89"
server_key_iv: "53c7b2deb056b5a449230f7565e06131"
server_key_version: "encryptionkey@0"
I20260621 05:10:18.548198 3980 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782018616648578-3598-0/minicluster-data/ts-2/wal/instance:
uuid: "0e0c3767fccc47b4ae51d70fb78bfb23"
format_stamp: "Formatted at 2026-06-21 05:10:18 on dist-test-slave-vrq1"
server_key: "a039e6e989e2a8733524568be3cf1d89"
server_key_iv: "53c7b2deb056b5a449230f7565e06131"
server_key_version: "encryptionkey@0"
I20260621 05:10:18.549813 3980 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260621 05:10:18.550660 3995 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 05:10:18.550909 3980 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260621 05:10:18.551009 3980 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782018616648578-3598-0/minicluster-data/ts-2/data,/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782018616648578-3598-0/minicluster-data/ts-2/wal
uuid: "0e0c3767fccc47b4ae51d70fb78bfb23"
format_stamp: "Formatted at 2026-06-21 05:10:18 on dist-test-slave-vrq1"
server_key: "a039e6e989e2a8733524568be3cf1d89"
server_key_iv: "53c7b2deb056b5a449230f7565e06131"
server_key_version: "encryptionkey@0"
I20260621 05:10:18.551084 3980 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782018616648578-3598-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782018616648578-3598-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782018616648578-3598-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 05:10:18.563969 3980 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 05:10:18.566330 3980 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 05:10:18.566524 3980 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 05:10:18.566802 3980 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 05:10:18.567121 3980 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 05:10:18.567154 3980 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 05:10:18.567226 3980 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 05:10:18.567260 3980 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 05:10:18.573547 3980 rpc_server.cc:307] RPC server started. Bound to: 127.3.131.131:40821
I20260621 05:10:18.573593 4108 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.131.131:40821 every 8 connection(s)
I20260621 05:10:18.573935 3980 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782018616648578-3598-0/minicluster-data/ts-2/data/info.pb
I20260621 05:10:18.579349 3598 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWZOx2_/build/release/bin/kudu as pid 3980
I20260621 05:10:18.579445 3598 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782018616648578-3598-0/minicluster-data/ts-2/wal/instance
I20260621 05:10:18.579670 3598 external_mini_cluster.cc:1428] Setting key 8a13ccc3a3c882591f0e7ca1c9e537a3
Jun 21 05:10:18 dist-test-slave-vrq1 krb5kdc[3605](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782018618, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.131.131@KRBTEST.COM for kudu/127.3.131.190@KRBTEST.COM
I20260621 05:10:18.585839 3708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:10:18.574740 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.131.131:39375 (local address 127.3.131.190:41035)
0621 05:10:18.574789 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0621 05:10:18.574791 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:10:18.574988 (+ 197us) server_negotiation.cc:407] Connection header received
0621 05:10:18.575498 (+ 510us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:10:18.575499 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:10:18.575511 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:10:18.575577 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:10:18.576556 (+ 979us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:10:18.577054 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:10:18.578189 (+ 1135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:10:18.578451 (+ 262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:10:18.578538 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:10:18.581872 (+ 3334us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:10:18.581873 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:10:18.581874 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:10:18.581882 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:10:18.583893 (+ 2011us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:10:18.584481 (+ 588us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:10:18.584482 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:10:18.584483 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:10:18.584624 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:10:18.584964 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:10:18.584965 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:10:18.584965 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:10:18.585108 (+ 143us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:10:18.585167 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0621 05:10:18.585640 (+ 473us) server_negotiation.cc:299] Negotiation successful
0621 05:10:18.585759 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260621 05:10:18.585994 4111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:10:18.574667 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.131.190:41035 (local address 127.3.131.131:39375)
0621 05:10:18.574835 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0621 05:10:18.574878 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0621 05:10:18.575386 (+ 508us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:10:18.575634 (+ 248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:10:18.575638 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:10:18.576231 (+ 593us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:10:18.576430 (+ 199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:10:18.576435 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:10:18.577156 (+ 721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:10:18.577157 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:10:18.578032 (+ 875us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:10:18.578035 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:10:18.578574 (+ 539us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:10:18.578575 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:10:18.578704 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:10:18.579579 (+ 875us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 05:10:18.579588 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:10:18.581647 (+ 2059us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 05:10:18.584045 (+ 2398us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:10:18.584056 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:10:18.584059 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:10:18.584357 (+ 298us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:10:18.584729 (+ 372us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:10:18.584730 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:10:18.584730 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:10:18.584877 (+ 147us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:10:18.585193 (+ 316us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 05:10:18.585200 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 05:10:18.585456 (+ 256us) client_negotiation.cc:769] Sending connection context
0621 05:10:18.585575 (+ 119us) client_negotiation.cc:240] Negotiation successful
0621 05:10:18.585762 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":81,"threads_started":1}
I20260621 05:10:18.586803 4109 heartbeater.cc:347] Connected to a master server at 127.3.131.190:41035
I20260621 05:10:18.586915 4109 heartbeater.cc:464] Registering TS with master...
I20260621 05:10:18.587080 4109 heartbeater.cc:511] Master 127.3.131.190:41035 requested a full tablet report, sending...
I20260621 05:10:18.587568 3650 ts_manager.cc:194] Registered new tserver with Master: 0e0c3767fccc47b4ae51d70fb78bfb23 (127.3.131.131:40821)
I20260621 05:10:18.588258 3650 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.131.131@KRBTEST.COM'} at 127.3.131.131:39375
I20260621 05:10:18.592713 3598 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 21 05:10:18 dist-test-slave-vrq1 krb5kdc[3605](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782018618, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 21 05:10:18 dist-test-slave-vrq1 krb5kdc[3605](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782018618, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.3.131.129@KRBTEST.COM
I20260621 05:10:18.621503 4116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:10:18.611293 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60486 (local address 127.3.131.129:35613)
0621 05:10:18.611546 (+ 253us) server_negotiation.cc:206] Beginning negotiation
0621 05:10:18.611552 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 05:10:18.611564 (+ 12us) server_negotiation.cc:407] Connection header received
0621 05:10:18.611593 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:10:18.611599 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:10:18.611615 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:10:18.611690 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:10:18.612532 (+ 842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:10:18.613291 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:10:18.614313 (+ 1022us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:10:18.614697 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:10:18.614773 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:10:18.617927 (+ 3154us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:10:18.617937 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:10:18.617938 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:10:18.617946 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:10:18.619773 (+ 1827us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:10:18.620298 (+ 525us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:10:18.620302 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:10:18.620302 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:10:18.620371 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:10:18.620689 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:10:18.620690 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:10:18.620690 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:10:18.620869 (+ 179us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:10:18.621029 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0621 05:10:18.621212 (+ 183us) server_negotiation.cc:299] Negotiation successful
0621 05:10:18.621357 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":216,"thread_start_us":106,"threads_started":1}
W20260621 05:10:18.621908 3755 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:60486
Jun 21 05:10:18 dist-test-slave-vrq1 krb5kdc[3605](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782018618, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 21 05:10:18 dist-test-slave-vrq1 krb5kdc[3605](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782018618, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.131.129@KRBTEST.COM
I20260621 05:10:18.647234 4116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:10:18.638923 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60490 (local address 127.3.131.129:35613)
0621 05:10:18.639104 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0621 05:10:18.639106 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:10:18.639118 (+ 12us) server_negotiation.cc:407] Connection header received
0621 05:10:18.639147 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:10:18.639149 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:10:18.639164 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:10:18.639315 (+ 151us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:10:18.639948 (+ 633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:10:18.640454 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:10:18.641276 (+ 822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:10:18.641486 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:10:18.641547 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:10:18.644353 (+ 2806us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:10:18.644354 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:10:18.644355 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:10:18.644363 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:10:18.645926 (+ 1563us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:10:18.646396 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:10:18.646397 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:10:18.646397 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:10:18.646462 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:10:18.646714 (+ 252us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:10:18.646714 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:10:18.646714 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:10:18.646816 (+ 102us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:10:18.646909 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0621 05:10:18.647040 (+ 131us) server_negotiation.cc:299] Negotiation successful
0621 05:10:18.647108 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":150}
I20260621 05:10:18.647905 3598 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWZOx2_/build/release/bin/kudu with pid 3712
I20260621 05:10:18.652732 3598 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWZOx2_/build/release/bin/kudu with pid 3846
I20260621 05:10:18.658074 3598 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWZOx2_/build/release/bin/kudu with pid 3980
I20260621 05:10:18.663061 3598 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWZOx2_/build/release/bin/kudu with pid 3618
2026-06-21T05:10:18Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (2022 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782018616648578-3598-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 21 05:10:18 dist-test-slave-vrq1 krb5kdc[4123](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 21 05:10:18 dist-test-slave-vrq1 krb5kdc[4123](info): set up 2 sockets
Jun 21 05:10:18 dist-test-slave-vrq1 krb5kdc[4123](info): commencing operation
krb5kdc: starting...
W20260621 05:10:19.739293 3598 mini_kdc.cc:121] Time spent starting KDC: real 1.054s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782018616648578-3598-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782018616648578-3598-0/krb5kdc/test-user.keytab.
Jun 21 05:10:19 dist-test-slave-vrq1 krb5kdc[4123](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782018619, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-21T05:10:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-21T05:10:19Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.131.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.131.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.131.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782018616648578-3598-0/krb5kdc/kudu_127.3.131.190.keytab.
Entry for principal kudu/127.3.131.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782018616648578-3598-0/krb5kdc/kudu_127.3.131.190.keytab.
I20260621 05:10:19.866611 3598 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWZOx2_/build/release/bin/kudu
/tmp/dist-test-taskWZOx2_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782018616648578-3598-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782018616648578-3598-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782018616648578-3598-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782018616648578-3598-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.131.190:37011
--webserver_interface=127.3.131.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782018616648578-3598-0/krb5kdc/kudu_127.3.131.190.keytab
--principal=kudu/127.3.131.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.131.148:42845
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.131.190:37011
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782018616648578-3598-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782018616648578-3598-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782018616648578-3598-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 05:10:19.949496 4136 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 05:10:19.949705 4136 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 05:10:19.949729 4136 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 05:10:19.950899 4136 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 05:10:19.950946 4136 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 05:10:19.950960 4136 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 05:10:19.950973 4136 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 05:10:19.950985 4136 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 05:10:19.952505 4136 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.131.148:42845
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782018616648578-3598-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782018616648578-3598-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.131.190:37011
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782018616648578-3598-0/krb5kdc/kudu_127.3.131.190.keytab
--principal=kudu/127.3.131.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.131.190:37011
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782018616648578-3598-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.131.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4136
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782018616648578-3598-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 21 Jun 2026 04:16:16 UTC on ccycloud.quasar-xycaha.root.comops.site
build id 4457881
I20260621 05:10:19.952764 4136 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 05:10:19.953015 4136 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 05:10:19.958202 4141 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:10:19.958250 4142 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:10:19.958250 4144 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:10:19.958271 4136 server_base.cc:1034] running on GCE node
I20260621 05:10:19.958735 4136 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 05:10:19.959074 4136 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 05:10:19.960273 4136 hybrid_clock.cc:648] HybridClock initialized: now 1782018619960256 us; error 68 us; skew 500 ppm
Jun 21 05:10:19 dist-test-slave-vrq1 krb5kdc[4123](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782018619, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.131.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 05:10:19.963153 4136 init.cc:377] Logged in from keytab as kudu/127.3.131.190@KRBTEST.COM (short username kudu)
I20260621 05:10:19.963775 4136 webserver.cc:460] Webserver started at http://127.3.131.190:35055/ using document root <none> and password file <none>
I20260621 05:10:19.964071 4136 fs_manager.cc:359] Metadata directory not provided
I20260621 05:10:19.964126 4136 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 05:10:19.964273 4136 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 05:10:19.965445 4136 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782018616648578-3598-0/minicluster-data/master-0/data/instance:
uuid: "aea57cda615f40f19590b9c06cef7e44"
format_stamp: "Formatted at 2026-06-21 05:10:19 on dist-test-slave-vrq1"
server_key: "504d3b731ce4f4730d98f2ecdd196abd"
server_key_iv: "a9d9d9379c5d0e2bc539452645198d06"
server_key_version: "encryptionkey@0"
I20260621 05:10:19.965847 4136 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782018616648578-3598-0/minicluster-data/master-0/wal/instance:
uuid: "aea57cda615f40f19590b9c06cef7e44"
format_stamp: "Formatted at 2026-06-21 05:10:19 on dist-test-slave-vrq1"
server_key: "504d3b731ce4f4730d98f2ecdd196abd"
server_key_iv: "a9d9d9379c5d0e2bc539452645198d06"
server_key_version: "encryptionkey@0"
I20260621 05:10:19.967573 4136 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260621 05:10:19.968633 4151 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 05:10:19.968884 4136 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260621 05:10:19.969022 4136 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782018616648578-3598-0/minicluster-data/master-0/data,/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782018616648578-3598-0/minicluster-data/master-0/wal
uuid: "aea57cda615f40f19590b9c06cef7e44"
format_stamp: "Formatted at 2026-06-21 05:10:19 on dist-test-slave-vrq1"
server_key: "504d3b731ce4f4730d98f2ecdd196abd"
server_key_iv: "a9d9d9379c5d0e2bc539452645198d06"
server_key_version: "encryptionkey@0"
I20260621 05:10:19.969156 4136 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782018616648578-3598-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782018616648578-3598-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782018616648578-3598-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 05:10:19.987946 4136 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 05:10:19.990332 4136 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 05:10:19.990520 4136 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 05:10:19.994926 4136 rpc_server.cc:307] RPC server started. Bound to: 127.3.131.190:37011
I20260621 05:10:19.994956 4203 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.131.190:37011 every 8 connection(s)
I20260621 05:10:19.995349 4136 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782018616648578-3598-0/minicluster-data/master-0/data/info.pb
I20260621 05:10:19.996366 4204 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 05:10:19.998857 4204 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aea57cda615f40f19590b9c06cef7e44: Bootstrap starting.
I20260621 05:10:19.999526 4204 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P aea57cda615f40f19590b9c06cef7e44: Neither blocks nor log segments found. Creating new log.
I20260621 05:10:19.999792 4204 log.cc:824] T 00000000000000000000000000000000 P aea57cda615f40f19590b9c06cef7e44: Log is configured to *not* fsync() on all Append() calls
I20260621 05:10:20.000550 4204 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aea57cda615f40f19590b9c06cef7e44: No bootstrap required, opened a new log
I20260621 05:10:20.002271 4204 raft_consensus.cc:348] T 00000000000000000000000000000000 P aea57cda615f40f19590b9c06cef7e44 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aea57cda615f40f19590b9c06cef7e44" member_type: VOTER last_known_addr { host: "127.3.131.190" port: 37011 } }
I20260621 05:10:20.002430 3598 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWZOx2_/build/release/bin/kudu as pid 4136
I20260621 05:10:20.002481 4204 raft_consensus.cc:374] T 00000000000000000000000000000000 P aea57cda615f40f19590b9c06cef7e44 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 05:10:20.002508 3598 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782018616648578-3598-0/minicluster-data/master-0/wal/instance
I20260621 05:10:20.002521 4204 raft_consensus.cc:729] T 00000000000000000000000000000000 P aea57cda615f40f19590b9c06cef7e44 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aea57cda615f40f19590b9c06cef7e44, State: Initialized, Role: FOLLOWER
I20260621 05:10:20.002674 3598 external_mini_cluster.cc:1428] Setting key 7a67115936cede5927b2d8c6f7334097
I20260621 05:10:20.002684 4204 consensus_queue.cc:260] T 00000000000000000000000000000000 P aea57cda615f40f19590b9c06cef7e44 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aea57cda615f40f19590b9c06cef7e44" member_type: VOTER last_known_addr { host: "127.3.131.190" port: 37011 } }
I20260621 05:10:20.002779 4204 raft_consensus.cc:388] T 00000000000000000000000000000000 P aea57cda615f40f19590b9c06cef7e44 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260621 05:10:20.002822 4204 raft_consensus.cc:482] T 00000000000000000000000000000000 P aea57cda615f40f19590b9c06cef7e44 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260621 05:10:20.002908 4204 raft_consensus.cc:3037] T 00000000000000000000000000000000 P aea57cda615f40f19590b9c06cef7e44 [term 0 FOLLOWER]: Advancing to term 1
I20260621 05:10:20.004194 4204 raft_consensus.cc:504] T 00000000000000000000000000000000 P aea57cda615f40f19590b9c06cef7e44 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aea57cda615f40f19590b9c06cef7e44" member_type: VOTER last_known_addr { host: "127.3.131.190" port: 37011 } }
I20260621 05:10:20.004292 4204 leader_election.cc:302] T 00000000000000000000000000000000 P aea57cda615f40f19590b9c06cef7e44 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: aea57cda615f40f19590b9c06cef7e44; no voters:
I20260621 05:10:20.004472 4204 leader_election.cc:288] T 00000000000000000000000000000000 P aea57cda615f40f19590b9c06cef7e44 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260621 05:10:20.004523 4209 raft_consensus.cc:2781] T 00000000000000000000000000000000 P aea57cda615f40f19590b9c06cef7e44 [term 1 FOLLOWER]: Leader election won for term 1
I20260621 05:10:20.004690 4209 raft_consensus.cc:686] T 00000000000000000000000000000000 P aea57cda615f40f19590b9c06cef7e44 [term 1 LEADER]: Becoming Leader. State: Replica: aea57cda615f40f19590b9c06cef7e44, State: Running, Role: LEADER
I20260621 05:10:20.004756 4204 sys_catalog.cc:564] T 00000000000000000000000000000000 P aea57cda615f40f19590b9c06cef7e44 [sys.catalog]: configured and running, proceeding with master startup.
I20260621 05:10:20.004787 4209 consensus_queue.cc:237] T 00000000000000000000000000000000 P aea57cda615f40f19590b9c06cef7e44 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aea57cda615f40f19590b9c06cef7e44" member_type: VOTER last_known_addr { host: "127.3.131.190" port: 37011 } }
I20260621 05:10:20.005187 4210 sys_catalog.cc:455] T 00000000000000000000000000000000 P aea57cda615f40f19590b9c06cef7e44 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "aea57cda615f40f19590b9c06cef7e44" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aea57cda615f40f19590b9c06cef7e44" member_type: VOTER last_known_addr { host: "127.3.131.190" port: 37011 } } }
I20260621 05:10:20.005285 4210 sys_catalog.cc:458] T 00000000000000000000000000000000 P aea57cda615f40f19590b9c06cef7e44 [sys.catalog]: This master's current role is: LEADER
I20260621 05:10:20.005518 4217 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260621 05:10:20.005609 4211 sys_catalog.cc:455] T 00000000000000000000000000000000 P aea57cda615f40f19590b9c06cef7e44 [sys.catalog]: SysCatalogTable state changed. Reason: New leader aea57cda615f40f19590b9c06cef7e44. Latest consensus state: current_term: 1 leader_uuid: "aea57cda615f40f19590b9c06cef7e44" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aea57cda615f40f19590b9c06cef7e44" member_type: VOTER last_known_addr { host: "127.3.131.190" port: 37011 } } }
I20260621 05:10:20.005730 4211 sys_catalog.cc:458] T 00000000000000000000000000000000 P aea57cda615f40f19590b9c06cef7e44 [sys.catalog]: This master's current role is: LEADER
I20260621 05:10:20.006016 4217 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260621 05:10:20.007824 4217 catalog_manager.cc:1347] Generated new cluster ID: f2cb5122480147b6a805a7829a7b3d00
I20260621 05:10:20.007885 4217 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Jun 21 05:10:20 dist-test-slave-vrq1 krb5kdc[4123](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782018619, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.131.190@KRBTEST.COM
I20260621 05:10:20.015020 4208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:10:20.003240 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48274 (local address 127.3.131.190:37011)
0621 05:10:20.003492 (+ 252us) server_negotiation.cc:206] Beginning negotiation
0621 05:10:20.003503 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0621 05:10:20.003551 (+ 48us) server_negotiation.cc:407] Connection header received
0621 05:10:20.003704 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:10:20.003714 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:10:20.003787 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:10:20.003952 (+ 165us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:10:20.004715 (+ 763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:10:20.006296 (+ 1581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:10:20.007290 (+ 994us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:10:20.007926 (+ 636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:10:20.008039 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:10:20.011057 (+ 3018us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:10:20.011062 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:10:20.011063 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:10:20.011077 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:10:20.013231 (+ 2154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:10:20.013692 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:10:20.013696 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:10:20.013696 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:10:20.013823 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:10:20.014055 (+ 232us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:10:20.014056 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:10:20.014056 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:10:20.014374 (+ 318us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:10:20.014518 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0621 05:10:20.014657 (+ 139us) server_negotiation.cc:299] Negotiation successful
0621 05:10:20.014794 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":193,"thread_start_us":100,"threads_started":1}
I20260621 05:10:20.027865 4217 catalog_manager.cc:1370] Generated new certificate authority record
I20260621 05:10:20.028414 4217 catalog_manager.cc:1504] Loading token signing keys...
I20260621 05:10:20.034269 4217 catalog_manager.cc:5868] T 00000000000000000000000000000000 P aea57cda615f40f19590b9c06cef7e44: Generated new TSK 0
I20260621 05:10:20.034483 4217 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.3.131.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.131.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.131.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782018616648578-3598-0/krb5kdc/kudu_127.3.131.129.keytab.
Entry for principal kudu/127.3.131.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782018616648578-3598-0/krb5kdc/kudu_127.3.131.129.keytab.
I20260621 05:10:20.062373 3598 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWZOx2_/build/release/bin/kudu
/tmp/dist-test-taskWZOx2_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782018616648578-3598-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782018616648578-3598-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782018616648578-3598-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782018616648578-3598-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.131.129:0
--local_ip_for_outbound_sockets=127.3.131.129
--webserver_interface=127.3.131.129
--webserver_port=0
--tserver_master_addrs=127.3.131.190:37011
--keytab_file=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782018616648578-3598-0/krb5kdc/kudu_127.3.131.129.keytab
--principal=kudu/127.3.131.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.131.148:42845
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782018616648578-3598-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782018616648578-3598-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782018616648578-3598-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 05:10:20.145632 4230 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 05:10:20.145890 4230 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 05:10:20.145934 4230 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 05:10:20.147202 4230 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 05:10:20.147274 4230 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 05:10:20.147320 4230 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.131.129
I20260621 05:10:20.148867 4230 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.131.148:42845
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782018616648578-3598-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782018616648578-3598-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782018616648578-3598-0/krb5kdc/kudu_127.3.131.129.keytab
--principal=kudu/127.3.131.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.131.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782018616648578-3598-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.131.129
--webserver_port=0
--tserver_master_addrs=127.3.131.190:37011
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4230
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.131.129
--log_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782018616648578-3598-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 21 Jun 2026 04:16:16 UTC on ccycloud.quasar-xycaha.root.comops.site
build id 4457881
I20260621 05:10:20.149156 4230 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 05:10:20.149410 4230 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 05:10:20.154527 4235 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:10:20.154500 4238 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:10:20.154516 4236 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:10:20.154587 4230 server_base.cc:1034] running on GCE node
I20260621 05:10:20.154961 4230 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 05:10:20.155253 4230 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 05:10:20.156445 4230 hybrid_clock.cc:648] HybridClock initialized: now 1782018620156423 us; error 46 us; skew 500 ppm
Jun 21 05:10:20 dist-test-slave-vrq1 krb5kdc[4123](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782018620, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.131.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 05:10:20.159085 4230 init.cc:377] Logged in from keytab as kudu/127.3.131.129@KRBTEST.COM (short username kudu)
I20260621 05:10:20.159613 4230 webserver.cc:460] Webserver started at http://127.3.131.129:46253/ using document root <none> and password file <none>
I20260621 05:10:20.159819 4230 fs_manager.cc:359] Metadata directory not provided
I20260621 05:10:20.159862 4230 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 05:10:20.159962 4230 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 05:10:20.160962 4230 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782018616648578-3598-0/minicluster-data/ts-0/data/instance:
uuid: "39270700cc3848e7873d58a446914485"
format_stamp: "Formatted at 2026-06-21 05:10:20 on dist-test-slave-vrq1"
server_key: "89e526ac6d25395074ceb92ebe4fb4ab"
server_key_iv: "ff8b232c786ea89f26e23d824dd9517a"
server_key_version: "encryptionkey@0"
I20260621 05:10:20.161273 4230 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782018616648578-3598-0/minicluster-data/ts-0/wal/instance:
uuid: "39270700cc3848e7873d58a446914485"
format_stamp: "Formatted at 2026-06-21 05:10:20 on dist-test-slave-vrq1"
server_key: "89e526ac6d25395074ceb92ebe4fb4ab"
server_key_iv: "ff8b232c786ea89f26e23d824dd9517a"
server_key_version: "encryptionkey@0"
I20260621 05:10:20.162563 4230 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260621 05:10:20.163381 4245 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 05:10:20.163565 4230 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260621 05:10:20.163664 4230 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782018616648578-3598-0/minicluster-data/ts-0/data,/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782018616648578-3598-0/minicluster-data/ts-0/wal
uuid: "39270700cc3848e7873d58a446914485"
format_stamp: "Formatted at 2026-06-21 05:10:20 on dist-test-slave-vrq1"
server_key: "89e526ac6d25395074ceb92ebe4fb4ab"
server_key_iv: "ff8b232c786ea89f26e23d824dd9517a"
server_key_version: "encryptionkey@0"
I20260621 05:10:20.163722 4230 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782018616648578-3598-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782018616648578-3598-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782018616648578-3598-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 05:10:20.174662 4230 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 05:10:20.177314 4230 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 05:10:20.177453 4230 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 05:10:20.177747 4230 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 05:10:20.178050 4230 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 05:10:20.178083 4230 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 05:10:20.178108 4230 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 05:10:20.178122 4230 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 05:10:20.184547 4230 rpc_server.cc:307] RPC server started. Bound to: 127.3.131.129:38917
I20260621 05:10:20.184588 4358 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.131.129:38917 every 8 connection(s)
I20260621 05:10:20.184993 4230 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782018616648578-3598-0/minicluster-data/ts-0/data/info.pb
I20260621 05:10:20.187855 3598 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWZOx2_/build/release/bin/kudu as pid 4230
I20260621 05:10:20.187932 3598 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782018616648578-3598-0/minicluster-data/ts-0/wal/instance
I20260621 05:10:20.188134 3598 external_mini_cluster.cc:1428] Setting key a3cf0c86470f137a5ee4930494659e81
Jun 21 05:10:20 dist-test-slave-vrq1 krb5kdc[4123](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782018620, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.131.129@KRBTEST.COM for kudu/127.3.131.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.3.131.130@KRBTEST.COM; defaulting to no policy
I20260621 05:10:20.196723 4208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:10:20.186005 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.131.129:52343 (local address 127.3.131.190:37011)
0621 05:10:20.186126 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0621 05:10:20.186128 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:10:20.186279 (+ 151us) server_negotiation.cc:407] Connection header received
0621 05:10:20.187433 (+ 1154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:10:20.187435 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:10:20.187447 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:10:20.187517 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:10:20.188340 (+ 823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:10:20.189091 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:10:20.190327 (+ 1236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:10:20.190558 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:10:20.190623 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:10:20.193452 (+ 2829us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:10:20.193454 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:10:20.193454 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:10:20.193464 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:10:20.195019 (+ 1555us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:10:20.195616 (+ 597us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:10:20.195617 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:10:20.195618 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:10:20.195683 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:10:20.196009 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:10:20.196010 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:10:20.196011 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:10:20.196137 (+ 126us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:10:20.196211 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0621 05:10:20.196517 (+ 306us) server_negotiation.cc:299] Negotiation successful
0621 05:10:20.196618 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260621 05:10:20.196928 4361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:10:20.185924 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.131.190:37011 (local address 127.3.131.129:52343)
0621 05:10:20.186112 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0621 05:10:20.186156 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0621 05:10:20.187310 (+ 1154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:10:20.187596 (+ 286us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:10:20.187599 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:10:20.187883 (+ 284us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:10:20.188149 (+ 266us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:10:20.188154 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:10:20.189192 (+ 1038us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:10:20.189193 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:10:20.190209 (+ 1016us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:10:20.190212 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:10:20.190648 (+ 436us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:10:20.190649 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:10:20.190808 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:10:20.191553 (+ 745us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 05:10:20.191564 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:10:20.193330 (+ 1766us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 05:10:20.195242 (+ 1912us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:10:20.195250 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:10:20.195253 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:10:20.195506 (+ 253us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:10:20.195810 (+ 304us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:10:20.195811 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:10:20.195811 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:10:20.195934 (+ 123us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:10:20.196219 (+ 285us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 05:10:20.196227 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 05:10:20.196377 (+ 150us) client_negotiation.cc:769] Sending connection context
0621 05:10:20.196482 (+ 105us) client_negotiation.cc:240] Negotiation successful
0621 05:10:20.196659 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":89,"threads_started":1}
I20260621 05:10:20.197590 4359 heartbeater.cc:347] Connected to a master server at 127.3.131.190:37011
I20260621 05:10:20.197731 4359 heartbeater.cc:464] Registering TS with master...
I20260621 05:10:20.197906 4359 heartbeater.cc:511] Master 127.3.131.190:37011 requested a full tablet report, sending...
I20260621 05:10:20.198464 4168 ts_manager.cc:194] Registered new tserver with Master: 39270700cc3848e7873d58a446914485 (127.3.131.129:38917)
I20260621 05:10:20.199150 4168 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.131.129@KRBTEST.COM'} at 127.3.131.129:52343
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.131.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.131.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782018616648578-3598-0/krb5kdc/kudu_127.3.131.130.keytab.
Entry for principal kudu/127.3.131.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782018616648578-3598-0/krb5kdc/kudu_127.3.131.130.keytab.
I20260621 05:10:20.218889 3598 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWZOx2_/build/release/bin/kudu
/tmp/dist-test-taskWZOx2_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782018616648578-3598-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782018616648578-3598-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782018616648578-3598-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782018616648578-3598-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.131.130:0
--local_ip_for_outbound_sockets=127.3.131.130
--webserver_interface=127.3.131.130
--webserver_port=0
--tserver_master_addrs=127.3.131.190:37011
--keytab_file=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782018616648578-3598-0/krb5kdc/kudu_127.3.131.130.keytab
--principal=kudu/127.3.131.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.131.148:42845
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782018616648578-3598-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782018616648578-3598-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782018616648578-3598-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 05:10:20.310858 4364 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 05:10:20.311088 4364 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 05:10:20.311113 4364 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 05:10:20.312369 4364 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 05:10:20.312417 4364 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 05:10:20.312445 4364 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.131.130
I20260621 05:10:20.313874 4364 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.131.148:42845
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782018616648578-3598-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782018616648578-3598-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782018616648578-3598-0/krb5kdc/kudu_127.3.131.130.keytab
--principal=kudu/127.3.131.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.131.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782018616648578-3598-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.131.130
--webserver_port=0
--tserver_master_addrs=127.3.131.190:37011
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4364
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.131.130
--log_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782018616648578-3598-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 21 Jun 2026 04:16:16 UTC on ccycloud.quasar-xycaha.root.comops.site
build id 4457881
I20260621 05:10:20.314126 4364 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 05:10:20.314330 4364 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 05:10:20.319523 4369 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:10:20.319532 4370 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:10:20.319679 4364 server_base.cc:1034] running on GCE node
W20260621 05:10:20.319523 4372 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:10:20.319965 4364 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 05:10:20.320232 4364 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 05:10:20.321417 4364 hybrid_clock.cc:648] HybridClock initialized: now 1782018620321397 us; error 34 us; skew 500 ppm
Jun 21 05:10:20 dist-test-slave-vrq1 krb5kdc[4123](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782018620, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.131.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 05:10:20.324319 4364 init.cc:377] Logged in from keytab as kudu/127.3.131.130@KRBTEST.COM (short username kudu)
I20260621 05:10:20.324781 4364 webserver.cc:460] Webserver started at http://127.3.131.130:45157/ using document root <none> and password file <none>
I20260621 05:10:20.324973 4364 fs_manager.cc:359] Metadata directory not provided
I20260621 05:10:20.325016 4364 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 05:10:20.325109 4364 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 05:10:20.326171 4364 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782018616648578-3598-0/minicluster-data/ts-1/data/instance:
uuid: "b971ebabde164f7293a93b3bf6fa98c1"
format_stamp: "Formatted at 2026-06-21 05:10:20 on dist-test-slave-vrq1"
server_key: "62edb2ef3f71f9cccff9911f41ec46b8"
server_key_iv: "3265e862569263e7dea317c28aa9f04e"
server_key_version: "encryptionkey@0"
I20260621 05:10:20.326478 4364 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782018616648578-3598-0/minicluster-data/ts-1/wal/instance:
uuid: "b971ebabde164f7293a93b3bf6fa98c1"
format_stamp: "Formatted at 2026-06-21 05:10:20 on dist-test-slave-vrq1"
server_key: "62edb2ef3f71f9cccff9911f41ec46b8"
server_key_iv: "3265e862569263e7dea317c28aa9f04e"
server_key_version: "encryptionkey@0"
I20260621 05:10:20.327940 4364 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.002s
I20260621 05:10:20.328749 4379 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 05:10:20.329030 4364 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260621 05:10:20.329161 4364 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782018616648578-3598-0/minicluster-data/ts-1/data,/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782018616648578-3598-0/minicluster-data/ts-1/wal
uuid: "b971ebabde164f7293a93b3bf6fa98c1"
format_stamp: "Formatted at 2026-06-21 05:10:20 on dist-test-slave-vrq1"
server_key: "62edb2ef3f71f9cccff9911f41ec46b8"
server_key_iv: "3265e862569263e7dea317c28aa9f04e"
server_key_version: "encryptionkey@0"
I20260621 05:10:20.329250 4364 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782018616648578-3598-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782018616648578-3598-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782018616648578-3598-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 05:10:20.355333 4364 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 05:10:20.357897 4364 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 05:10:20.358098 4364 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 05:10:20.358420 4364 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 05:10:20.358764 4364 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 05:10:20.358825 4364 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 05:10:20.358868 4364 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 05:10:20.358894 4364 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 05:10:20.365207 4364 rpc_server.cc:307] RPC server started. Bound to: 127.3.131.130:45067
I20260621 05:10:20.365295 4492 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.131.130:45067 every 8 connection(s)
I20260621 05:10:20.365851 4364 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782018616648578-3598-0/minicluster-data/ts-1/data/info.pb
Jun 21 05:10:20 dist-test-slave-vrq1 krb5kdc[4123](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782018620, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.131.130@KRBTEST.COM for kudu/127.3.131.190@KRBTEST.COM
I20260621 05:10:20.375145 3598 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWZOx2_/build/release/bin/kudu as pid 4364
I20260621 05:10:20.375346 3598 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782018616648578-3598-0/minicluster-data/ts-1/wal/instance
I20260621 05:10:20.375545 3598 external_mini_cluster.cc:1428] Setting key 48c798c5155bd3e6e5d3bb356bc66c92
I20260621 05:10:20.377254 4208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:10:20.366769 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.131.130:36127 (local address 127.3.131.190:37011)
0621 05:10:20.366813 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0621 05:10:20.366816 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0621 05:10:20.367166 (+ 350us) server_negotiation.cc:407] Connection header received
0621 05:10:20.368388 (+ 1222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:10:20.368389 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:10:20.368401 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:10:20.368468 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:10:20.369139 (+ 671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:10:20.369644 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:10:20.370785 (+ 1141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:10:20.370982 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:10:20.371044 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:10:20.373815 (+ 2771us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:10:20.373817 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:10:20.373817 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:10:20.373825 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:10:20.375442 (+ 1617us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:10:20.376046 (+ 604us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:10:20.376048 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:10:20.376048 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:10:20.376121 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:10:20.376430 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:10:20.376431 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:10:20.376431 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:10:20.376567 (+ 136us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:10:20.376652 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0621 05:10:20.376971 (+ 319us) server_negotiation.cc:299] Negotiation successful
0621 05:10:20.377074 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260621 05:10:20.377537 4495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:10:20.366753 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.131.190:37011 (local address 127.3.131.130:36127)
0621 05:10:20.366931 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0621 05:10:20.366980 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0621 05:10:20.368253 (+ 1273us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:10:20.368552 (+ 299us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:10:20.368556 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:10:20.368838 (+ 282us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:10:20.369048 (+ 210us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:10:20.369053 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:10:20.369742 (+ 689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:10:20.369744 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:10:20.370696 (+ 952us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:10:20.370698 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:10:20.371092 (+ 394us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:10:20.371094 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:10:20.371249 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:10:20.371971 (+ 722us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 05:10:20.371980 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:10:20.373703 (+ 1723us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 05:10:20.375552 (+ 1849us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:10:20.375562 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:10:20.375565 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:10:20.375951 (+ 386us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:10:20.376191 (+ 240us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:10:20.376192 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:10:20.376193 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:10:20.376372 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:10:20.376653 (+ 281us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 05:10:20.376661 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 05:10:20.376827 (+ 166us) client_negotiation.cc:769] Sending connection context
0621 05:10:20.376937 (+ 110us) client_negotiation.cc:240] Negotiation successful
0621 05:10:20.377255 (+ 318us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":102,"threads_started":1}
I20260621 05:10:20.378638 4493 heartbeater.cc:347] Connected to a master server at 127.3.131.190:37011
I20260621 05:10:20.378782 4493 heartbeater.cc:464] Registering TS with master...
I20260621 05:10:20.379006 4493 heartbeater.cc:511] Master 127.3.131.190:37011 requested a full tablet report, sending...
I20260621 05:10:20.379505 4168 ts_manager.cc:194] Registered new tserver with Master: b971ebabde164f7293a93b3bf6fa98c1 (127.3.131.130:45067)
I20260621 05:10:20.380105 4168 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.131.130@KRBTEST.COM'} at 127.3.131.130:36127
WARNING: no policy specified for kudu/127.3.131.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.131.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.131.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782018616648578-3598-0/krb5kdc/kudu_127.3.131.131.keytab.
Entry for principal kudu/127.3.131.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782018616648578-3598-0/krb5kdc/kudu_127.3.131.131.keytab.
I20260621 05:10:20.403945 3598 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWZOx2_/build/release/bin/kudu
/tmp/dist-test-taskWZOx2_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782018616648578-3598-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782018616648578-3598-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782018616648578-3598-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782018616648578-3598-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.131.131:0
--local_ip_for_outbound_sockets=127.3.131.131
--webserver_interface=127.3.131.131
--webserver_port=0
--tserver_master_addrs=127.3.131.190:37011
--keytab_file=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782018616648578-3598-0/krb5kdc/kudu_127.3.131.131.keytab
--principal=kudu/127.3.131.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.131.148:42845
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782018616648578-3598-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782018616648578-3598-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782018616648578-3598-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 05:10:20.497228 4498 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 05:10:20.497463 4498 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 05:10:20.497490 4498 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 05:10:20.498808 4498 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 05:10:20.498853 4498 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 05:10:20.498880 4498 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.131.131
I20260621 05:10:20.500442 4498 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.131.148:42845
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782018616648578-3598-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782018616648578-3598-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782018616648578-3598-0/krb5kdc/kudu_127.3.131.131.keytab
--principal=kudu/127.3.131.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.131.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782018616648578-3598-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.131.131
--webserver_port=0
--tserver_master_addrs=127.3.131.190:37011
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4498
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.131.131
--log_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782018616648578-3598-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 21 Jun 2026 04:16:16 UTC on ccycloud.quasar-xycaha.root.comops.site
build id 4457881
I20260621 05:10:20.500718 4498 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 05:10:20.500982 4498 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 05:10:20.506490 4503 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:10:20.506511 4506 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:10:20.506500 4504 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:10:20.506723 4498 server_base.cc:1034] running on GCE node
I20260621 05:10:20.506996 4498 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 05:10:20.507350 4498 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 05:10:20.508602 4498 hybrid_clock.cc:648] HybridClock initialized: now 1782018620508552 us; error 75 us; skew 500 ppm
Jun 21 05:10:20 dist-test-slave-vrq1 krb5kdc[4123](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782018620, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.131.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 05:10:20.511493 4498 init.cc:377] Logged in from keytab as kudu/127.3.131.131@KRBTEST.COM (short username kudu)
I20260621 05:10:20.512152 4498 webserver.cc:460] Webserver started at http://127.3.131.131:41629/ using document root <none> and password file <none>
I20260621 05:10:20.512393 4498 fs_manager.cc:359] Metadata directory not provided
I20260621 05:10:20.512470 4498 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 05:10:20.512625 4498 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 05:10:20.513800 4498 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782018616648578-3598-0/minicluster-data/ts-2/data/instance:
uuid: "c20f1c390da54d79b8d7899a540acd1f"
format_stamp: "Formatted at 2026-06-21 05:10:20 on dist-test-slave-vrq1"
server_key: "dec1ac06ed9c5c72cef26fc7bd6b3049"
server_key_iv: "31cfefa051085af0614a4ccad65e0612"
server_key_version: "encryptionkey@0"
I20260621 05:10:20.514206 4498 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782018616648578-3598-0/minicluster-data/ts-2/wal/instance:
uuid: "c20f1c390da54d79b8d7899a540acd1f"
format_stamp: "Formatted at 2026-06-21 05:10:20 on dist-test-slave-vrq1"
server_key: "dec1ac06ed9c5c72cef26fc7bd6b3049"
server_key_iv: "31cfefa051085af0614a4ccad65e0612"
server_key_version: "encryptionkey@0"
I20260621 05:10:20.515805 4498 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260621 05:10:20.516747 4513 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 05:10:20.516954 4498 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260621 05:10:20.517063 4498 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782018616648578-3598-0/minicluster-data/ts-2/data,/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782018616648578-3598-0/minicluster-data/ts-2/wal
uuid: "c20f1c390da54d79b8d7899a540acd1f"
format_stamp: "Formatted at 2026-06-21 05:10:20 on dist-test-slave-vrq1"
server_key: "dec1ac06ed9c5c72cef26fc7bd6b3049"
server_key_iv: "31cfefa051085af0614a4ccad65e0612"
server_key_version: "encryptionkey@0"
I20260621 05:10:20.517148 4498 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782018616648578-3598-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782018616648578-3598-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782018616648578-3598-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 05:10:20.532876 4498 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 05:10:20.535727 4498 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 05:10:20.535882 4498 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 05:10:20.536128 4498 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 05:10:20.536468 4498 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 05:10:20.536501 4498 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 05:10:20.536527 4498 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 05:10:20.536573 4498 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 05:10:20.542625 4498 rpc_server.cc:307] RPC server started. Bound to: 127.3.131.131:35493
I20260621 05:10:20.542709 4626 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.131.131:35493 every 8 connection(s)
I20260621 05:10:20.543113 4498 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782018616648578-3598-0/minicluster-data/ts-2/data/info.pb
I20260621 05:10:20.550266 3598 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWZOx2_/build/release/bin/kudu as pid 4498
I20260621 05:10:20.550357 3598 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782018616648578-3598-0/minicluster-data/ts-2/wal/instance
Jun 21 05:10:20 dist-test-slave-vrq1 krb5kdc[4123](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782018620, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.131.131@KRBTEST.COM for kudu/127.3.131.190@KRBTEST.COM
I20260621 05:10:20.550539 3598 external_mini_cluster.cc:1428] Setting key f4eb862cc7b67658e4d845ed97411a63
I20260621 05:10:20.554632 4208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:10:20.544049 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.131.131:51987 (local address 127.3.131.190:37011)
0621 05:10:20.544102 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0621 05:10:20.544105 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0621 05:10:20.544341 (+ 236us) server_negotiation.cc:407] Connection header received
0621 05:10:20.545355 (+ 1014us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:10:20.545357 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:10:20.545367 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:10:20.545429 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:10:20.546140 (+ 711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:10:20.546674 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:10:20.547848 (+ 1174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:10:20.548044 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:10:20.548116 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:10:20.551078 (+ 2962us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:10:20.551079 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:10:20.551080 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:10:20.551090 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:10:20.552835 (+ 1745us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:10:20.553427 (+ 592us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:10:20.553428 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:10:20.553431 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:10:20.553512 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:10:20.553859 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:10:20.553860 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:10:20.553861 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:10:20.553990 (+ 129us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:10:20.554055 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0621 05:10:20.554451 (+ 396us) server_negotiation.cc:299] Negotiation successful
0621 05:10:20.554550 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260621 05:10:20.554719 4629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:10:20.544048 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.131.190:37011 (local address 127.3.131.131:51987)
0621 05:10:20.544211 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0621 05:10:20.544249 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0621 05:10:20.545215 (+ 966us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:10:20.545512 (+ 297us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:10:20.545515 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:10:20.545804 (+ 289us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:10:20.546005 (+ 201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:10:20.546009 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:10:20.546784 (+ 775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:10:20.546785 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:10:20.547751 (+ 966us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:10:20.547754 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:10:20.548149 (+ 395us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:10:20.548150 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:10:20.548328 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:10:20.549106 (+ 778us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 05:10:20.549115 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:10:20.550918 (+ 1803us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 05:10:20.552943 (+ 2025us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:10:20.552955 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:10:20.552958 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:10:20.553303 (+ 345us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:10:20.553648 (+ 345us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:10:20.553649 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:10:20.553649 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:10:20.553783 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:10:20.554114 (+ 331us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 05:10:20.554121 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 05:10:20.554278 (+ 157us) client_negotiation.cc:769] Sending connection context
0621 05:10:20.554382 (+ 104us) client_negotiation.cc:240] Negotiation successful
0621 05:10:20.554532 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":90,"threads_started":1}
I20260621 05:10:20.555634 4627 heartbeater.cc:347] Connected to a master server at 127.3.131.190:37011
I20260621 05:10:20.555754 4627 heartbeater.cc:464] Registering TS with master...
I20260621 05:10:20.555922 4627 heartbeater.cc:511] Master 127.3.131.190:37011 requested a full tablet report, sending...
I20260621 05:10:20.556381 4168 ts_manager.cc:194] Registered new tserver with Master: c20f1c390da54d79b8d7899a540acd1f (127.3.131.131:35493)
I20260621 05:10:20.556839 4168 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.131.131@KRBTEST.COM'} at 127.3.131.131:51987
I20260621 05:10:20.562924 3598 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260621 05:10:20.571906 4208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:10:20.564843 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48276 (local address 127.3.131.190:37011)
0621 05:10:20.564924 (+ 81us) server_negotiation.cc:206] Beginning negotiation
0621 05:10:20.564926 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:10:20.565043 (+ 117us) server_negotiation.cc:407] Connection header received
0621 05:10:20.565118 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:10:20.565119 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:10:20.565129 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:10:20.565186 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:10:20.565870 (+ 684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:10:20.566425 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:10:20.567354 (+ 929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:10:20.567551 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:10:20.567615 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:10:20.568867 (+ 1252us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:10:20.568869 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:10:20.568869 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:10:20.568876 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:10:20.570478 (+ 1602us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:10:20.570868 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:10:20.570869 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:10:20.570869 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:10:20.570931 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:10:20.571329 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:10:20.571330 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:10:20.571330 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:10:20.571494 (+ 164us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:10:20.571579 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0621 05:10:20.571749 (+ 170us) server_negotiation.cc:299] Negotiation successful
0621 05:10:20.571815 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260621 05:10:20.574735 4168 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48276:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260621 05:10:20.576424 4168 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260621 05:10:20.583660 4638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:10:20.579296 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.131.131:35493 (local address 127.0.0.1:60642)
0621 05:10:20.579580 (+ 284us) negotiation.cc:107] Waiting for socket to connect
0621 05:10:20.579630 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0621 05:10:20.579702 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:10:20.580086 (+ 384us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:10:20.580087 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:10:20.580089 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 05:10:20.580144 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:10:20.580146 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:10:20.581052 (+ 906us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:10:20.581053 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:10:20.582473 (+ 1420us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:10:20.582475 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:10:20.583291 (+ 816us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:10:20.583292 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:10:20.583385 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:10:20.583401 (+ 16us) client_negotiation.cc:769] Sending connection context
0621 05:10:20.583498 (+ 97us) client_negotiation.cc:240] Negotiation successful
0621 05:10:20.583559 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":256,"mutex_wait_us":1,"thread_start_us":92,"threads_started":1}
I20260621 05:10:20.583961 4640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:10:20.579330 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47708 (local address 127.3.131.130:45067)
0621 05:10:20.579698 (+ 368us) server_negotiation.cc:206] Beginning negotiation
0621 05:10:20.579703 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 05:10:20.579857 (+ 154us) server_negotiation.cc:407] Connection header received
0621 05:10:20.579895 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:10:20.579897 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:10:20.579909 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:10:20.580139 (+ 230us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 05:10:20.580350 (+ 211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:10:20.581403 (+ 1053us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:10:20.582547 (+ 1144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:10:20.583367 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:10:20.583444 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:10:20.583507 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0621 05:10:20.583846 (+ 339us) server_negotiation.cc:299] Negotiation successful
0621 05:10:20.583891 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":330,"thread_start_us":87,"threads_started":1}
I20260621 05:10:20.583774 4643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:10:20.579581 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48510 (local address 127.3.131.129:38917)
0621 05:10:20.579809 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0621 05:10:20.579813 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 05:10:20.579825 (+ 12us) server_negotiation.cc:407] Connection header received
0621 05:10:20.579867 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:10:20.579869 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:10:20.579887 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:10:20.579971 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 05:10:20.580196 (+ 225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:10:20.581370 (+ 1174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:10:20.582667 (+ 1297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:10:20.583396 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:10:20.583462 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:10:20.583511 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0621 05:10:20.583634 (+ 123us) server_negotiation.cc:299] Negotiation successful
0621 05:10:20.583688 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":190,"thread_start_us":91,"threads_started":1}
I20260621 05:10:20.584142 4641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:10:20.579400 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60642 (local address 127.3.131.131:35493)
0621 05:10:20.579855 (+ 455us) server_negotiation.cc:206] Beginning negotiation
0621 05:10:20.579859 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 05:10:20.579868 (+ 9us) server_negotiation.cc:407] Connection header received
0621 05:10:20.579905 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:10:20.579908 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:10:20.579924 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:10:20.579991 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 05:10:20.580223 (+ 232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:10:20.580941 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:10:20.582574 (+ 1633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:10:20.583123 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:10:20.583288 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:10:20.583358 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0621 05:10:20.583899 (+ 541us) server_negotiation.cc:299] Negotiation successful
0621 05:10:20.583992 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":421,"thread_start_us":85,"threads_started":1}
I20260621 05:10:20.583793 4642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:10:20.579471 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.131.129:38917 (local address 127.0.0.1:48510)
0621 05:10:20.579595 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0621 05:10:20.579633 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0621 05:10:20.579703 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:10:20.579987 (+ 284us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:10:20.579990 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:10:20.579992 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 05:10:20.580085 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:10:20.580087 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:10:20.581497 (+ 1410us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:10:20.581498 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:10:20.582581 (+ 1083us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:10:20.582582 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:10:20.583475 (+ 893us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:10:20.583476 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:10:20.583534 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:10:20.583537 (+ 3us) client_negotiation.cc:769] Sending connection context
0621 05:10:20.583605 (+ 68us) client_negotiation.cc:240] Negotiation successful
0621 05:10:20.583644 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":107,"mutex_wait_us":31,"thread_start_us":128,"threads_started":1}
I20260621 05:10:20.583806 4639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:10:20.579229 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.131.130:45067 (local address 127.0.0.1:47708)
0621 05:10:20.579738 (+ 509us) negotiation.cc:107] Waiting for socket to connect
0621 05:10:20.579748 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0621 05:10:20.579795 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:10:20.580225 (+ 430us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:10:20.580225 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:10:20.580228 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 05:10:20.580271 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:10:20.580273 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:10:20.581509 (+ 1236us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:10:20.581510 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:10:20.582453 (+ 943us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:10:20.582455 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:10:20.583551 (+ 1096us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:10:20.583552 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:10:20.583622 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:10:20.583626 (+ 4us) client_negotiation.cc:769] Sending connection context
0621 05:10:20.583679 (+ 53us) client_negotiation.cc:240] Negotiation successful
0621 05:10:20.583720 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":261,"thread_start_us":60,"threads_started":1}
I20260621 05:10:20.585255 4561 tablet_service.cc:1467] Processing CreateTablet for tablet 44ad36a106fc4b61aafa0c8627ffebfb (DEFAULT_TABLE table=test-table [id=c5b4e4c2f881494da0304ab4ab99fcdf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 05:10:20.585363 4293 tablet_service.cc:1467] Processing CreateTablet for tablet 44ad36a106fc4b61aafa0c8627ffebfb (DEFAULT_TABLE table=test-table [id=c5b4e4c2f881494da0304ab4ab99fcdf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 05:10:20.585570 4427 tablet_service.cc:1467] Processing CreateTablet for tablet 44ad36a106fc4b61aafa0c8627ffebfb (DEFAULT_TABLE table=test-table [id=c5b4e4c2f881494da0304ab4ab99fcdf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 05:10:20.585731 4561 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 44ad36a106fc4b61aafa0c8627ffebfb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 05:10:20.585749 4293 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 44ad36a106fc4b61aafa0c8627ffebfb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 05:10:20.585897 4427 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 44ad36a106fc4b61aafa0c8627ffebfb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 05:10:20.589071 4644 tablet_bootstrap.cc:492] T 44ad36a106fc4b61aafa0c8627ffebfb P 39270700cc3848e7873d58a446914485: Bootstrap starting.
I20260621 05:10:20.589602 4646 tablet_bootstrap.cc:492] T 44ad36a106fc4b61aafa0c8627ffebfb P b971ebabde164f7293a93b3bf6fa98c1: Bootstrap starting.
I20260621 05:10:20.590376 4644 tablet_bootstrap.cc:654] T 44ad36a106fc4b61aafa0c8627ffebfb P 39270700cc3848e7873d58a446914485: Neither blocks nor log segments found. Creating new log.
I20260621 05:10:20.590375 4646 tablet_bootstrap.cc:654] T 44ad36a106fc4b61aafa0c8627ffebfb P b971ebabde164f7293a93b3bf6fa98c1: Neither blocks nor log segments found. Creating new log.
I20260621 05:10:20.591094 4644 log.cc:824] T 44ad36a106fc4b61aafa0c8627ffebfb P 39270700cc3848e7873d58a446914485: Log is configured to *not* fsync() on all Append() calls
I20260621 05:10:20.591094 4646 log.cc:824] T 44ad36a106fc4b61aafa0c8627ffebfb P b971ebabde164f7293a93b3bf6fa98c1: Log is configured to *not* fsync() on all Append() calls
I20260621 05:10:20.591740 4645 tablet_bootstrap.cc:492] T 44ad36a106fc4b61aafa0c8627ffebfb P c20f1c390da54d79b8d7899a540acd1f: Bootstrap starting.
I20260621 05:10:20.592051 4646 tablet_bootstrap.cc:492] T 44ad36a106fc4b61aafa0c8627ffebfb P b971ebabde164f7293a93b3bf6fa98c1: No bootstrap required, opened a new log
I20260621 05:10:20.592140 4646 ts_tablet_manager.cc:1397] T 44ad36a106fc4b61aafa0c8627ffebfb P b971ebabde164f7293a93b3bf6fa98c1: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260621 05:10:20.592188 4644 tablet_bootstrap.cc:492] T 44ad36a106fc4b61aafa0c8627ffebfb P 39270700cc3848e7873d58a446914485: No bootstrap required, opened a new log
I20260621 05:10:20.592264 4644 ts_tablet_manager.cc:1397] T 44ad36a106fc4b61aafa0c8627ffebfb P 39270700cc3848e7873d58a446914485: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260621 05:10:20.592499 4645 tablet_bootstrap.cc:654] T 44ad36a106fc4b61aafa0c8627ffebfb P c20f1c390da54d79b8d7899a540acd1f: Neither blocks nor log segments found. Creating new log.
I20260621 05:10:20.593135 4645 log.cc:824] T 44ad36a106fc4b61aafa0c8627ffebfb P c20f1c390da54d79b8d7899a540acd1f: Log is configured to *not* fsync() on all Append() calls
I20260621 05:10:20.594101 4645 tablet_bootstrap.cc:492] T 44ad36a106fc4b61aafa0c8627ffebfb P c20f1c390da54d79b8d7899a540acd1f: No bootstrap required, opened a new log
I20260621 05:10:20.594201 4645 ts_tablet_manager.cc:1397] T 44ad36a106fc4b61aafa0c8627ffebfb P c20f1c390da54d79b8d7899a540acd1f: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260621 05:10:20.594156 4646 raft_consensus.cc:348] T 44ad36a106fc4b61aafa0c8627ffebfb P b971ebabde164f7293a93b3bf6fa98c1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b971ebabde164f7293a93b3bf6fa98c1" member_type: VOTER last_known_addr { host: "127.3.131.130" port: 45067 } } peers { permanent_uuid: "39270700cc3848e7873d58a446914485" member_type: VOTER last_known_addr { host: "127.3.131.129" port: 38917 } } peers { permanent_uuid: "c20f1c390da54d79b8d7899a540acd1f" member_type: VOTER last_known_addr { host: "127.3.131.131" port: 35493 } }
I20260621 05:10:20.594354 4646 raft_consensus.cc:374] T 44ad36a106fc4b61aafa0c8627ffebfb P b971ebabde164f7293a93b3bf6fa98c1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 05:10:20.594398 4646 raft_consensus.cc:729] T 44ad36a106fc4b61aafa0c8627ffebfb P b971ebabde164f7293a93b3bf6fa98c1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b971ebabde164f7293a93b3bf6fa98c1, State: Initialized, Role: FOLLOWER
I20260621 05:10:20.594516 4646 consensus_queue.cc:260] T 44ad36a106fc4b61aafa0c8627ffebfb P b971ebabde164f7293a93b3bf6fa98c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b971ebabde164f7293a93b3bf6fa98c1" member_type: VOTER last_known_addr { host: "127.3.131.130" port: 45067 } } peers { permanent_uuid: "39270700cc3848e7873d58a446914485" member_type: VOTER last_known_addr { host: "127.3.131.129" port: 38917 } } peers { permanent_uuid: "c20f1c390da54d79b8d7899a540acd1f" member_type: VOTER last_known_addr { host: "127.3.131.131" port: 35493 } }
I20260621 05:10:20.594537 4644 raft_consensus.cc:348] T 44ad36a106fc4b61aafa0c8627ffebfb P 39270700cc3848e7873d58a446914485 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b971ebabde164f7293a93b3bf6fa98c1" member_type: VOTER last_known_addr { host: "127.3.131.130" port: 45067 } } peers { permanent_uuid: "39270700cc3848e7873d58a446914485" member_type: VOTER last_known_addr { host: "127.3.131.129" port: 38917 } } peers { permanent_uuid: "c20f1c390da54d79b8d7899a540acd1f" member_type: VOTER last_known_addr { host: "127.3.131.131" port: 35493 } }
I20260621 05:10:20.594756 4644 raft_consensus.cc:374] T 44ad36a106fc4b61aafa0c8627ffebfb P 39270700cc3848e7873d58a446914485 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 05:10:20.594810 4644 raft_consensus.cc:729] T 44ad36a106fc4b61aafa0c8627ffebfb P 39270700cc3848e7873d58a446914485 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 39270700cc3848e7873d58a446914485, State: Initialized, Role: FOLLOWER
I20260621 05:10:20.594846 4646 ts_tablet_manager.cc:1428] T 44ad36a106fc4b61aafa0c8627ffebfb P b971ebabde164f7293a93b3bf6fa98c1: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260621 05:10:20.594978 4493 heartbeater.cc:503] Master 127.3.131.190:37011 was elected leader, sending a full tablet report...
I20260621 05:10:20.594985 4644 consensus_queue.cc:260] T 44ad36a106fc4b61aafa0c8627ffebfb P 39270700cc3848e7873d58a446914485 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b971ebabde164f7293a93b3bf6fa98c1" member_type: VOTER last_known_addr { host: "127.3.131.130" port: 45067 } } peers { permanent_uuid: "39270700cc3848e7873d58a446914485" member_type: VOTER last_known_addr { host: "127.3.131.129" port: 38917 } } peers { permanent_uuid: "c20f1c390da54d79b8d7899a540acd1f" member_type: VOTER last_known_addr { host: "127.3.131.131" port: 35493 } }
I20260621 05:10:20.595394 4644 ts_tablet_manager.cc:1428] T 44ad36a106fc4b61aafa0c8627ffebfb P 39270700cc3848e7873d58a446914485: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260621 05:10:20.595552 4359 heartbeater.cc:503] Master 127.3.131.190:37011 was elected leader, sending a full tablet report...
I20260621 05:10:20.596422 4645 raft_consensus.cc:348] T 44ad36a106fc4b61aafa0c8627ffebfb P c20f1c390da54d79b8d7899a540acd1f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b971ebabde164f7293a93b3bf6fa98c1" member_type: VOTER last_known_addr { host: "127.3.131.130" port: 45067 } } peers { permanent_uuid: "39270700cc3848e7873d58a446914485" member_type: VOTER last_known_addr { host: "127.3.131.129" port: 38917 } } peers { permanent_uuid: "c20f1c390da54d79b8d7899a540acd1f" member_type: VOTER last_known_addr { host: "127.3.131.131" port: 35493 } }
I20260621 05:10:20.596587 4645 raft_consensus.cc:374] T 44ad36a106fc4b61aafa0c8627ffebfb P c20f1c390da54d79b8d7899a540acd1f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 05:10:20.596645 4645 raft_consensus.cc:729] T 44ad36a106fc4b61aafa0c8627ffebfb P c20f1c390da54d79b8d7899a540acd1f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c20f1c390da54d79b8d7899a540acd1f, State: Initialized, Role: FOLLOWER
I20260621 05:10:20.596760 4645 consensus_queue.cc:260] T 44ad36a106fc4b61aafa0c8627ffebfb P c20f1c390da54d79b8d7899a540acd1f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b971ebabde164f7293a93b3bf6fa98c1" member_type: VOTER last_known_addr { host: "127.3.131.130" port: 45067 } } peers { permanent_uuid: "39270700cc3848e7873d58a446914485" member_type: VOTER last_known_addr { host: "127.3.131.129" port: 38917 } } peers { permanent_uuid: "c20f1c390da54d79b8d7899a540acd1f" member_type: VOTER last_known_addr { host: "127.3.131.131" port: 35493 } }
I20260621 05:10:20.597059 4645 ts_tablet_manager.cc:1428] T 44ad36a106fc4b61aafa0c8627ffebfb P c20f1c390da54d79b8d7899a540acd1f: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260621 05:10:20.597119 4627 heartbeater.cc:503] Master 127.3.131.190:37011 was elected leader, sending a full tablet report...
W20260621 05:10:20.616621 4494 tablet.cc:2369] T 44ad36a106fc4b61aafa0c8627ffebfb P b971ebabde164f7293a93b3bf6fa98c1: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260621 05:10:20.685818 4360 tablet.cc:2369] T 44ad36a106fc4b61aafa0c8627ffebfb P 39270700cc3848e7873d58a446914485: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260621 05:10:20.793860 4628 tablet.cc:2369] T 44ad36a106fc4b61aafa0c8627ffebfb P c20f1c390da54d79b8d7899a540acd1f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260621 05:10:20.952255 4652 raft_consensus.cc:482] T 44ad36a106fc4b61aafa0c8627ffebfb P c20f1c390da54d79b8d7899a540acd1f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260621 05:10:20.952441 4652 raft_consensus.cc:504] T 44ad36a106fc4b61aafa0c8627ffebfb P c20f1c390da54d79b8d7899a540acd1f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b971ebabde164f7293a93b3bf6fa98c1" member_type: VOTER last_known_addr { host: "127.3.131.130" port: 45067 } } peers { permanent_uuid: "39270700cc3848e7873d58a446914485" member_type: VOTER last_known_addr { host: "127.3.131.129" port: 38917 } } peers { permanent_uuid: "c20f1c390da54d79b8d7899a540acd1f" member_type: VOTER last_known_addr { host: "127.3.131.131" port: 35493 } }
I20260621 05:10:20.952839 4652 leader_election.cc:288] T 44ad36a106fc4b61aafa0c8627ffebfb P c20f1c390da54d79b8d7899a540acd1f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b971ebabde164f7293a93b3bf6fa98c1 (127.3.131.130:45067), 39270700cc3848e7873d58a446914485 (127.3.131.129:38917)
I20260621 05:10:20.956444 4643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:10:20.953397 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.131.131:60433 (local address 127.3.131.129:38917)
0621 05:10:20.953477 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0621 05:10:20.953480 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0621 05:10:20.953577 (+ 97us) server_negotiation.cc:407] Connection header received
0621 05:10:20.953619 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:10:20.953620 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:10:20.953635 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:10:20.953696 (+ 61us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 05:10:20.953892 (+ 196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:10:20.954460 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:10:20.955694 (+ 1234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:10:20.956061 (+ 367us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:10:20.956123 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:10:20.956130 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0621 05:10:20.956323 (+ 193us) server_negotiation.cc:299] Negotiation successful
0621 05:10:20.956369 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260621 05:10:20.956459 4653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:10:20.953316 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.131.129:38917 (local address 127.3.131.131:60433)
0621 05:10:20.953500 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0621 05:10:20.953511 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0621 05:10:20.953575 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:10:20.953720 (+ 145us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:10:20.953721 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:10:20.953722 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 05:10:20.953795 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:10:20.953797 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:10:20.954578 (+ 781us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:10:20.954579 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:10:20.955579 (+ 1000us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:10:20.955582 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:10:20.956157 (+ 575us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:10:20.956158 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:10:20.956249 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:10:20.956253 (+ 4us) client_negotiation.cc:769] Sending connection context
0621 05:10:20.956299 (+ 46us) client_negotiation.cc:240] Negotiation successful
0621 05:10:20.956353 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":163,"mutex_wait_us":1,"thread_start_us":127,"threads_started":1}
I20260621 05:10:20.956517 4640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:10:20.953192 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.131.131:42237 (local address 127.3.131.130:45067)
0621 05:10:20.953246 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0621 05:10:20.953248 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:10:20.953325 (+ 77us) server_negotiation.cc:407] Connection header received
0621 05:10:20.953415 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:10:20.953416 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:10:20.953430 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:10:20.953487 (+ 57us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 05:10:20.953725 (+ 238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:10:20.954276 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:10:20.955660 (+ 1384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:10:20.956080 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:10:20.956184 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:10:20.956193 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0621 05:10:20.956381 (+ 188us) server_negotiation.cc:299] Negotiation successful
0621 05:10:20.956431 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260621 05:10:20.956459 4629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:10:20.953111 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.131.130:45067 (local address 127.3.131.131:42237)
0621 05:10:20.953174 (+ 63us) negotiation.cc:107] Waiting for socket to connect
0621 05:10:20.953187 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0621 05:10:20.953317 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:10:20.953511 (+ 194us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:10:20.953512 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:10:20.953513 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 05:10:20.953615 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:10:20.953618 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:10:20.954377 (+ 759us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:10:20.954378 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:10:20.955553 (+ 1175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:10:20.955557 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:10:20.956199 (+ 642us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:10:20.956200 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:10:20.956273 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:10:20.956276 (+ 3us) client_negotiation.cc:769] Sending connection context
0621 05:10:20.956351 (+ 75us) client_negotiation.cc:240] Negotiation successful
0621 05:10:20.956390 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":39,"mutex_wait_us":77}
I20260621 05:10:20.956948 4313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44ad36a106fc4b61aafa0c8627ffebfb" candidate_uuid: "c20f1c390da54d79b8d7899a540acd1f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39270700cc3848e7873d58a446914485" is_pre_election: true
I20260621 05:10:20.957019 4447 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44ad36a106fc4b61aafa0c8627ffebfb" candidate_uuid: "c20f1c390da54d79b8d7899a540acd1f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b971ebabde164f7293a93b3bf6fa98c1" is_pre_election: true
I20260621 05:10:20.957142 4313 raft_consensus.cc:2454] T 44ad36a106fc4b61aafa0c8627ffebfb P 39270700cc3848e7873d58a446914485 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c20f1c390da54d79b8d7899a540acd1f in term 0.
I20260621 05:10:20.957178 4447 raft_consensus.cc:2454] T 44ad36a106fc4b61aafa0c8627ffebfb P b971ebabde164f7293a93b3bf6fa98c1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c20f1c390da54d79b8d7899a540acd1f in term 0.
I20260621 05:10:20.957415 4517 leader_election.cc:302] T 44ad36a106fc4b61aafa0c8627ffebfb P c20f1c390da54d79b8d7899a540acd1f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 39270700cc3848e7873d58a446914485, c20f1c390da54d79b8d7899a540acd1f; no voters:
I20260621 05:10:20.957578 4652 raft_consensus.cc:2781] T 44ad36a106fc4b61aafa0c8627ffebfb P c20f1c390da54d79b8d7899a540acd1f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260621 05:10:20.957674 4652 raft_consensus.cc:482] T 44ad36a106fc4b61aafa0c8627ffebfb P c20f1c390da54d79b8d7899a540acd1f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260621 05:10:20.957707 4652 raft_consensus.cc:3037] T 44ad36a106fc4b61aafa0c8627ffebfb P c20f1c390da54d79b8d7899a540acd1f [term 0 FOLLOWER]: Advancing to term 1
I20260621 05:10:20.958514 4652 raft_consensus.cc:504] T 44ad36a106fc4b61aafa0c8627ffebfb P c20f1c390da54d79b8d7899a540acd1f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b971ebabde164f7293a93b3bf6fa98c1" member_type: VOTER last_known_addr { host: "127.3.131.130" port: 45067 } } peers { permanent_uuid: "39270700cc3848e7873d58a446914485" member_type: VOTER last_known_addr { host: "127.3.131.129" port: 38917 } } peers { permanent_uuid: "c20f1c390da54d79b8d7899a540acd1f" member_type: VOTER last_known_addr { host: "127.3.131.131" port: 35493 } }
I20260621 05:10:20.958674 4652 leader_election.cc:288] T 44ad36a106fc4b61aafa0c8627ffebfb P c20f1c390da54d79b8d7899a540acd1f [CANDIDATE]: Term 1 election: Requested vote from peers b971ebabde164f7293a93b3bf6fa98c1 (127.3.131.130:45067), 39270700cc3848e7873d58a446914485 (127.3.131.129:38917)
I20260621 05:10:20.958932 4447 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44ad36a106fc4b61aafa0c8627ffebfb" candidate_uuid: "c20f1c390da54d79b8d7899a540acd1f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b971ebabde164f7293a93b3bf6fa98c1"
I20260621 05:10:20.958951 4313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44ad36a106fc4b61aafa0c8627ffebfb" candidate_uuid: "c20f1c390da54d79b8d7899a540acd1f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39270700cc3848e7873d58a446914485"
I20260621 05:10:20.959028 4447 raft_consensus.cc:3037] T 44ad36a106fc4b61aafa0c8627ffebfb P b971ebabde164f7293a93b3bf6fa98c1 [term 0 FOLLOWER]: Advancing to term 1
I20260621 05:10:20.959025 4313 raft_consensus.cc:3037] T 44ad36a106fc4b61aafa0c8627ffebfb P 39270700cc3848e7873d58a446914485 [term 0 FOLLOWER]: Advancing to term 1
I20260621 05:10:20.959813 4313 raft_consensus.cc:2454] T 44ad36a106fc4b61aafa0c8627ffebfb P 39270700cc3848e7873d58a446914485 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c20f1c390da54d79b8d7899a540acd1f in term 1.
I20260621 05:10:20.959887 4447 raft_consensus.cc:2454] T 44ad36a106fc4b61aafa0c8627ffebfb P b971ebabde164f7293a93b3bf6fa98c1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c20f1c390da54d79b8d7899a540acd1f in term 1.
I20260621 05:10:20.959985 4517 leader_election.cc:302] T 44ad36a106fc4b61aafa0c8627ffebfb P c20f1c390da54d79b8d7899a540acd1f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 39270700cc3848e7873d58a446914485, c20f1c390da54d79b8d7899a540acd1f; no voters:
I20260621 05:10:20.960105 4652 raft_consensus.cc:2781] T 44ad36a106fc4b61aafa0c8627ffebfb P c20f1c390da54d79b8d7899a540acd1f [term 1 FOLLOWER]: Leader election won for term 1
I20260621 05:10:20.960378 4652 raft_consensus.cc:686] T 44ad36a106fc4b61aafa0c8627ffebfb P c20f1c390da54d79b8d7899a540acd1f [term 1 LEADER]: Becoming Leader. State: Replica: c20f1c390da54d79b8d7899a540acd1f, State: Running, Role: LEADER
I20260621 05:10:20.960528 4652 consensus_queue.cc:237] T 44ad36a106fc4b61aafa0c8627ffebfb P c20f1c390da54d79b8d7899a540acd1f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b971ebabde164f7293a93b3bf6fa98c1" member_type: VOTER last_known_addr { host: "127.3.131.130" port: 45067 } } peers { permanent_uuid: "39270700cc3848e7873d58a446914485" member_type: VOTER last_known_addr { host: "127.3.131.129" port: 38917 } } peers { permanent_uuid: "c20f1c390da54d79b8d7899a540acd1f" member_type: VOTER last_known_addr { host: "127.3.131.131" port: 35493 } }
I20260621 05:10:20.961898 4167 catalog_manager.cc:5495] T 44ad36a106fc4b61aafa0c8627ffebfb P c20f1c390da54d79b8d7899a540acd1f reported cstate change: term changed from 0 to 1, leader changed from <none> to c20f1c390da54d79b8d7899a540acd1f (127.3.131.131). New cstate: current_term: 1 leader_uuid: "c20f1c390da54d79b8d7899a540acd1f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b971ebabde164f7293a93b3bf6fa98c1" member_type: VOTER last_known_addr { host: "127.3.131.130" port: 45067 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "39270700cc3848e7873d58a446914485" member_type: VOTER last_known_addr { host: "127.3.131.129" port: 38917 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c20f1c390da54d79b8d7899a540acd1f" member_type: VOTER last_known_addr { host: "127.3.131.131" port: 35493 } health_report { overall_health: HEALTHY } } }
Jun 21 05:10:21 dist-test-slave-vrq1 krb5kdc[4123](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782018619, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.131.129@KRBTEST.COM
I20260621 05:10:21.033702 4643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:10:21.025530 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48522 (local address 127.3.131.129:38917)
0621 05:10:21.025624 (+ 94us) server_negotiation.cc:206] Beginning negotiation
0621 05:10:21.025626 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:10:21.025729 (+ 103us) server_negotiation.cc:407] Connection header received
0621 05:10:21.025804 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:10:21.025805 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:10:21.025816 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:10:21.025868 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:10:21.026529 (+ 661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:10:21.027020 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:10:21.027842 (+ 822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:10:21.028043 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:10:21.028136 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:10:21.030539 (+ 2403us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:10:21.030560 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:10:21.030561 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:10:21.030568 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:10:21.032432 (+ 1864us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:10:21.032810 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:10:21.032813 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:10:21.032814 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:10:21.032886 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:10:21.033096 (+ 210us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:10:21.033097 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:10:21.033097 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:10:21.033288 (+ 191us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:10:21.033441 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0621 05:10:21.033481 (+ 40us) server_negotiation.cc:299] Negotiation successful
0621 05:10:21.033572 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
Jun 21 05:10:21 dist-test-slave-vrq1 krb5kdc[4123](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782018621, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 21 05:10:21 dist-test-slave-vrq1 krb5kdc[4123](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782018621, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.3.131.129@KRBTEST.COM
I20260621 05:10:21.058521 4643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:10:21.050199 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48532 (local address 127.3.131.129:38917)
0621 05:10:21.050323 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0621 05:10:21.050324 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0621 05:10:21.050341 (+ 17us) server_negotiation.cc:407] Connection header received
0621 05:10:21.050407 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:10:21.050408 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:10:21.050416 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:10:21.050481 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:10:21.051156 (+ 675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:10:21.051753 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:10:21.052553 (+ 800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:10:21.052762 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:10:21.052830 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:10:21.055552 (+ 2722us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:10:21.055554 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:10:21.055554 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:10:21.055563 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:10:21.057271 (+ 1708us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:10:21.057639 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:10:21.057640 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:10:21.057640 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:10:21.057699 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:10:21.057946 (+ 247us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:10:21.057946 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:10:21.057947 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:10:21.058050 (+ 103us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:10:21.058126 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0621 05:10:21.058319 (+ 193us) server_negotiation.cc:299] Negotiation successful
0621 05:10:21.058391 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91}
Jun 21 05:10:21 dist-test-slave-vrq1 krb5kdc[4123](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782018621, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 21 05:10:21 dist-test-slave-vrq1 krb5kdc[4123](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782018621, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.131.129@KRBTEST.COM
I20260621 05:10:21.083137 4643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:10:21.075320 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48534 (local address 127.3.131.129:38917)
0621 05:10:21.075386 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0621 05:10:21.075388 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:10:21.075400 (+ 12us) server_negotiation.cc:407] Connection header received
0621 05:10:21.075480 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:10:21.075481 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:10:21.075497 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:10:21.075562 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:10:21.076170 (+ 608us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:10:21.076681 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:10:21.077417 (+ 736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:10:21.077610 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:10:21.077688 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:10:21.080174 (+ 2486us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:10:21.080176 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:10:21.080176 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:10:21.080184 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:10:21.081670 (+ 1486us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:10:21.082167 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:10:21.082168 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:10:21.082169 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:10:21.082232 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:10:21.082514 (+ 282us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:10:21.082515 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:10:21.082516 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:10:21.082639 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:10:21.082735 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0621 05:10:21.082946 (+ 211us) server_negotiation.cc:299] Negotiation successful
0621 05:10:21.083022 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
W20260621 05:10:21.083693 4273 tablet_service.cc:2908] Rejecting scan request for tablet 44ad36a106fc4b61aafa0c8627ffebfb: Uninitialized: clean time has not yet been initialized
I20260621 05:10:21.084661 3598 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWZOx2_/build/release/bin/kudu with pid 4230
I20260621 05:10:21.089650 3598 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWZOx2_/build/release/bin/kudu with pid 4364
I20260621 05:10:21.095053 3598 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWZOx2_/build/release/bin/kudu with pid 4498
I20260621 05:10:21.099722 3598 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWZOx2_/build/release/bin/kudu with pid 4136
2026-06-21T05:10:21Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (2436 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782018616648578-3598-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 21 05:10:21 dist-test-slave-vrq1 krb5kdc[4668](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 21 05:10:21 dist-test-slave-vrq1 krb5kdc[4668](info): set up 2 sockets
Jun 21 05:10:21 dist-test-slave-vrq1 krb5kdc[4668](info): commencing operation
krb5kdc: starting...
W20260621 05:10:22.163105 3598 mini_kdc.cc:121] Time spent starting KDC: real 1.041s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782018616648578-3598-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782018616648578-3598-0/krb5kdc/test-user.keytab.
Jun 21 05:10:22 dist-test-slave-vrq1 krb5kdc[4668](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782018622, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-21T05:10:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-21T05:10:22Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.131.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.131.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.131.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782018616648578-3598-0/krb5kdc/kudu_127.3.131.190.keytab.
Entry for principal kudu/127.3.131.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782018616648578-3598-0/krb5kdc/kudu_127.3.131.190.keytab.
I20260621 05:10:22.303027 3598 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWZOx2_/build/release/bin/kudu
/tmp/dist-test-taskWZOx2_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782018616648578-3598-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782018616648578-3598-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782018616648578-3598-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782018616648578-3598-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.131.190:43199
--webserver_interface=127.3.131.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782018616648578-3598-0/krb5kdc/kudu_127.3.131.190.keytab
--principal=kudu/127.3.131.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.131.148:41101
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.131.190:43199
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782018616648578-3598-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782018616648578-3598-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782018616648578-3598-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 05:10:22.385708 4681 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 05:10:22.385959 4681 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 05:10:22.386005 4681 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 05:10:22.387221 4681 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 05:10:22.387286 4681 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260621 05:10:22.387307 4681 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 05:10:22.387326 4681 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 05:10:22.387358 4681 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 05:10:22.387380 4681 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 05:10:22.388878 4681 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.131.148:41101
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782018616648578-3598-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782018616648578-3598-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.131.190:43199
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782018616648578-3598-0/krb5kdc/kudu_127.3.131.190.keytab
--principal=kudu/127.3.131.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.131.190:43199
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782018616648578-3598-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.131.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4681
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782018616648578-3598-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 21 Jun 2026 04:16:16 UTC on ccycloud.quasar-xycaha.root.comops.site
build id 4457881
I20260621 05:10:22.389161 4681 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 05:10:22.389387 4681 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 05:10:22.394655 4689 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:10:22.394788 4687 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:10:22.394655 4686 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:10:22.394924 4681 server_base.cc:1034] running on GCE node
I20260621 05:10:22.395321 4681 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 05:10:22.395624 4681 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 05:10:22.396786 4681 hybrid_clock.cc:648] HybridClock initialized: now 1782018622396770 us; error 47 us; skew 500 ppm
Jun 21 05:10:22 dist-test-slave-vrq1 krb5kdc[4668](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782018622, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.131.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 05:10:22.404628 4681 init.cc:377] Logged in from keytab as kudu/127.3.131.190@KRBTEST.COM (short username kudu)
I20260621 05:10:22.405165 4681 webserver.cc:460] Webserver started at http://127.3.131.190:40739/ using document root <none> and password file <none>
I20260621 05:10:22.405421 4681 fs_manager.cc:359] Metadata directory not provided
I20260621 05:10:22.405472 4681 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 05:10:22.405627 4681 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 05:10:22.406759 4681 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782018616648578-3598-0/minicluster-data/master-0/data/instance:
uuid: "d11704c4279f426fa489c4af98674cd5"
format_stamp: "Formatted at 2026-06-21 05:10:22 on dist-test-slave-vrq1"
server_key: "353b0fe1ac072ab31f9e6f8f9af05d3d"
server_key_iv: "9eb9385a1385bf2108c5fb3ffacdde47"
server_key_version: "encryptionkey@0"
I20260621 05:10:22.407155 4681 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782018616648578-3598-0/minicluster-data/master-0/wal/instance:
uuid: "d11704c4279f426fa489c4af98674cd5"
format_stamp: "Formatted at 2026-06-21 05:10:22 on dist-test-slave-vrq1"
server_key: "353b0fe1ac072ab31f9e6f8f9af05d3d"
server_key_iv: "9eb9385a1385bf2108c5fb3ffacdde47"
server_key_version: "encryptionkey@0"
I20260621 05:10:22.408692 4681 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260621 05:10:22.409621 4696 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 05:10:22.409957 4681 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260621 05:10:22.410144 4681 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782018616648578-3598-0/minicluster-data/master-0/data,/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782018616648578-3598-0/minicluster-data/master-0/wal
uuid: "d11704c4279f426fa489c4af98674cd5"
format_stamp: "Formatted at 2026-06-21 05:10:22 on dist-test-slave-vrq1"
server_key: "353b0fe1ac072ab31f9e6f8f9af05d3d"
server_key_iv: "9eb9385a1385bf2108c5fb3ffacdde47"
server_key_version: "encryptionkey@0"
I20260621 05:10:22.410236 4681 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782018616648578-3598-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782018616648578-3598-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782018616648578-3598-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 05:10:22.424342 4681 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 05:10:22.426919 4681 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 05:10:22.427098 4681 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 05:10:22.431861 4681 rpc_server.cc:307] RPC server started. Bound to: 127.3.131.190:43199
I20260621 05:10:22.431941 4758 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.131.190:43199 every 8 connection(s)
I20260621 05:10:22.432305 4681 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782018616648578-3598-0/minicluster-data/master-0/data/info.pb
I20260621 05:10:22.433089 4759 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 05:10:22.435609 4759 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d11704c4279f426fa489c4af98674cd5: Bootstrap starting.
I20260621 05:10:22.436223 4759 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d11704c4279f426fa489c4af98674cd5: Neither blocks nor log segments found. Creating new log.
I20260621 05:10:22.436517 4759 log.cc:824] T 00000000000000000000000000000000 P d11704c4279f426fa489c4af98674cd5: Log is configured to *not* fsync() on all Append() calls
I20260621 05:10:22.437223 4759 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d11704c4279f426fa489c4af98674cd5: No bootstrap required, opened a new log
I20260621 05:10:22.438804 3598 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWZOx2_/build/release/bin/kudu as pid 4681
I20260621 05:10:22.438935 3598 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782018616648578-3598-0/minicluster-data/master-0/wal/instance
I20260621 05:10:22.439216 3598 external_mini_cluster.cc:1428] Setting key 1f1125cb862d009935b445a5b0da7717
I20260621 05:10:22.439992 4759 raft_consensus.cc:348] T 00000000000000000000000000000000 P d11704c4279f426fa489c4af98674cd5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d11704c4279f426fa489c4af98674cd5" member_type: VOTER last_known_addr { host: "127.3.131.190" port: 43199 } }
I20260621 05:10:22.440161 4759 raft_consensus.cc:374] T 00000000000000000000000000000000 P d11704c4279f426fa489c4af98674cd5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 05:10:22.440186 4759 raft_consensus.cc:729] T 00000000000000000000000000000000 P d11704c4279f426fa489c4af98674cd5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d11704c4279f426fa489c4af98674cd5, State: Initialized, Role: FOLLOWER
I20260621 05:10:22.440335 4759 consensus_queue.cc:260] T 00000000000000000000000000000000 P d11704c4279f426fa489c4af98674cd5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d11704c4279f426fa489c4af98674cd5" member_type: VOTER last_known_addr { host: "127.3.131.190" port: 43199 } }
I20260621 05:10:22.440397 4759 raft_consensus.cc:388] T 00000000000000000000000000000000 P d11704c4279f426fa489c4af98674cd5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260621 05:10:22.440441 4759 raft_consensus.cc:482] T 00000000000000000000000000000000 P d11704c4279f426fa489c4af98674cd5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260621 05:10:22.440522 4759 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d11704c4279f426fa489c4af98674cd5 [term 0 FOLLOWER]: Advancing to term 1
I20260621 05:10:22.441309 4759 raft_consensus.cc:504] T 00000000000000000000000000000000 P d11704c4279f426fa489c4af98674cd5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d11704c4279f426fa489c4af98674cd5" member_type: VOTER last_known_addr { host: "127.3.131.190" port: 43199 } }
I20260621 05:10:22.441399 4759 leader_election.cc:302] T 00000000000000000000000000000000 P d11704c4279f426fa489c4af98674cd5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d11704c4279f426fa489c4af98674cd5; no voters:
I20260621 05:10:22.441609 4759 leader_election.cc:288] T 00000000000000000000000000000000 P d11704c4279f426fa489c4af98674cd5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260621 05:10:22.441632 4764 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d11704c4279f426fa489c4af98674cd5 [term 1 FOLLOWER]: Leader election won for term 1
I20260621 05:10:22.441743 4764 raft_consensus.cc:686] T 00000000000000000000000000000000 P d11704c4279f426fa489c4af98674cd5 [term 1 LEADER]: Becoming Leader. State: Replica: d11704c4279f426fa489c4af98674cd5, State: Running, Role: LEADER
I20260621 05:10:22.441881 4759 sys_catalog.cc:564] T 00000000000000000000000000000000 P d11704c4279f426fa489c4af98674cd5 [sys.catalog]: configured and running, proceeding with master startup.
I20260621 05:10:22.441886 4764 consensus_queue.cc:237] T 00000000000000000000000000000000 P d11704c4279f426fa489c4af98674cd5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d11704c4279f426fa489c4af98674cd5" member_type: VOTER last_known_addr { host: "127.3.131.190" port: 43199 } }
I20260621 05:10:22.442341 4765 sys_catalog.cc:455] T 00000000000000000000000000000000 P d11704c4279f426fa489c4af98674cd5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d11704c4279f426fa489c4af98674cd5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d11704c4279f426fa489c4af98674cd5" member_type: VOTER last_known_addr { host: "127.3.131.190" port: 43199 } } }
I20260621 05:10:22.442420 4765 sys_catalog.cc:458] T 00000000000000000000000000000000 P d11704c4279f426fa489c4af98674cd5 [sys.catalog]: This master's current role is: LEADER
I20260621 05:10:22.442587 4766 sys_catalog.cc:455] T 00000000000000000000000000000000 P d11704c4279f426fa489c4af98674cd5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d11704c4279f426fa489c4af98674cd5. Latest consensus state: current_term: 1 leader_uuid: "d11704c4279f426fa489c4af98674cd5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d11704c4279f426fa489c4af98674cd5" member_type: VOTER last_known_addr { host: "127.3.131.190" port: 43199 } } }
I20260621 05:10:22.442651 4766 sys_catalog.cc:458] T 00000000000000000000000000000000 P d11704c4279f426fa489c4af98674cd5 [sys.catalog]: This master's current role is: LEADER
I20260621 05:10:22.442858 4771 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260621 05:10:22.443423 4771 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260621 05:10:22.445070 4771 catalog_manager.cc:1347] Generated new cluster ID: 3440f38818e14867bcb39d9c7168d09d
I20260621 05:10:22.445127 4771 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Jun 21 05:10:22 dist-test-slave-vrq1 krb5kdc[4668](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782018622, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.131.190@KRBTEST.COM
I20260621 05:10:22.451296 4763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:10:22.439873 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46752 (local address 127.3.131.190:43199)
0621 05:10:22.440057 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0621 05:10:22.440073 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0621 05:10:22.440099 (+ 26us) server_negotiation.cc:407] Connection header received
0621 05:10:22.440237 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:10:22.440245 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:10:22.440305 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:10:22.440476 (+ 171us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:10:22.441236 (+ 760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:10:22.442553 (+ 1317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:10:22.443761 (+ 1208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:10:22.444365 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:10:22.444518 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:10:22.447558 (+ 3040us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:10:22.447564 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:10:22.447564 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:10:22.447578 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:10:22.449588 (+ 2010us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:10:22.450009 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:10:22.450013 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:10:22.450013 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:10:22.450131 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:10:22.450347 (+ 216us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:10:22.450348 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:10:22.450348 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:10:22.450625 (+ 277us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:10:22.450774 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0621 05:10:22.450898 (+ 124us) server_negotiation.cc:299] Negotiation successful
0621 05:10:22.451063 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":128,"thread_start_us":99,"threads_started":1}
I20260621 05:10:22.452387 4771 catalog_manager.cc:1370] Generated new certificate authority record
I20260621 05:10:22.453052 4771 catalog_manager.cc:1504] Loading token signing keys...
I20260621 05:10:22.458591 4771 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d11704c4279f426fa489c4af98674cd5: Generated new TSK 0
I20260621 05:10:22.458820 4771 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.3.131.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.131.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.131.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782018616648578-3598-0/krb5kdc/kudu_127.3.131.129.keytab.
Entry for principal kudu/127.3.131.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782018616648578-3598-0/krb5kdc/kudu_127.3.131.129.keytab.
I20260621 05:10:22.496395 3598 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWZOx2_/build/release/bin/kudu
/tmp/dist-test-taskWZOx2_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782018616648578-3598-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782018616648578-3598-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782018616648578-3598-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782018616648578-3598-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.131.129:0
--local_ip_for_outbound_sockets=127.3.131.129
--webserver_interface=127.3.131.129
--webserver_port=0
--tserver_master_addrs=127.3.131.190:43199
--keytab_file=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782018616648578-3598-0/krb5kdc/kudu_127.3.131.129.keytab
--principal=kudu/127.3.131.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.131.148:41101
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782018616648578-3598-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782018616648578-3598-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782018616648578-3598-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 05:10:22.588781 4785 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 05:10:22.589017 4785 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 05:10:22.589047 4785 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 05:10:22.590356 4785 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 05:10:22.590411 4785 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 05:10:22.590435 4785 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260621 05:10:22.590454 4785 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.131.129
I20260621 05:10:22.592216 4785 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.131.148:41101
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782018616648578-3598-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782018616648578-3598-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782018616648578-3598-0/krb5kdc/kudu_127.3.131.129.keytab
--principal=kudu/127.3.131.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.131.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782018616648578-3598-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.131.129
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.3.131.190:43199
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4785
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.131.129
--log_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782018616648578-3598-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 21 Jun 2026 04:16:16 UTC on ccycloud.quasar-xycaha.root.comops.site
build id 4457881
I20260621 05:10:22.592491 4785 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 05:10:22.592757 4785 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260621 05:10:22.598824 4785 server_base.cc:1034] running on GCE node
W20260621 05:10:22.598791 4790 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:10:22.598948 4791 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:10:22.598904 4793 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:10:22.599376 4785 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 05:10:22.599654 4785 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 05:10:22.600880 4785 hybrid_clock.cc:648] HybridClock initialized: now 1782018622600840 us; error 41 us; skew 500 ppm
Jun 21 05:10:22 dist-test-slave-vrq1 krb5kdc[4668](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782018622, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.131.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 05:10:22.603932 4785 init.cc:377] Logged in from keytab as kudu/127.3.131.129@KRBTEST.COM (short username kudu)
I20260621 05:10:22.604460 4785 webserver.cc:460] Webserver started at http://127.3.131.129:44243/ using document root <none> and password file <none>
I20260621 05:10:22.604820 4785 fs_manager.cc:359] Metadata directory not provided
I20260621 05:10:22.604938 4785 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 05:10:22.605234 4785 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 05:10:22.606529 4785 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782018616648578-3598-0/minicluster-data/ts-0/data/instance:
uuid: "4f7fee4354ab4ed5b1bfae3ec9f66d4a"
format_stamp: "Formatted at 2026-06-21 05:10:22 on dist-test-slave-vrq1"
server_key: "df2bcfbd5226c8f23cd4f31ac7b8bf05"
server_key_iv: "edbd083488f445b02d9aa943f2ff82a4"
server_key_version: "encryptionkey@0"
I20260621 05:10:22.606961 4785 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782018616648578-3598-0/minicluster-data/ts-0/wal/instance:
uuid: "4f7fee4354ab4ed5b1bfae3ec9f66d4a"
format_stamp: "Formatted at 2026-06-21 05:10:22 on dist-test-slave-vrq1"
server_key: "df2bcfbd5226c8f23cd4f31ac7b8bf05"
server_key_iv: "edbd083488f445b02d9aa943f2ff82a4"
server_key_version: "encryptionkey@0"
I20260621 05:10:22.608888 4785 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.001s sys 0.000s
I20260621 05:10:22.609882 4800 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 05:10:22.610203 4785 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260621 05:10:22.610364 4785 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782018616648578-3598-0/minicluster-data/ts-0/data,/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782018616648578-3598-0/minicluster-data/ts-0/wal
uuid: "4f7fee4354ab4ed5b1bfae3ec9f66d4a"
format_stamp: "Formatted at 2026-06-21 05:10:22 on dist-test-slave-vrq1"
server_key: "df2bcfbd5226c8f23cd4f31ac7b8bf05"
server_key_iv: "edbd083488f445b02d9aa943f2ff82a4"
server_key_version: "encryptionkey@0"
I20260621 05:10:22.610457 4785 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782018616648578-3598-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782018616648578-3598-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782018616648578-3598-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 05:10:22.626853 4785 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 05:10:22.629491 4785 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 05:10:22.629712 4785 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 05:10:22.630303 4785 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 05:10:22.630368 4785 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 05:10:22.630404 4785 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 05:10:22.630441 4785 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 05:10:22.637800 4785 rpc_server.cc:307] RPC server started. Bound to: 127.3.131.129:41195
I20260621 05:10:22.637900 4915 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.131.129:41195 every 8 connection(s)
I20260621 05:10:22.638250 4785 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782018616648578-3598-0/minicluster-data/ts-0/data/info.pb
Jun 21 05:10:22 dist-test-slave-vrq1 krb5kdc[4668](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782018622, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.131.129@KRBTEST.COM for kudu/127.3.131.190@KRBTEST.COM
I20260621 05:10:22.642531 3598 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWZOx2_/build/release/bin/kudu as pid 4785
I20260621 05:10:22.642647 3598 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782018616648578-3598-0/minicluster-data/ts-0/wal/instance
I20260621 05:10:22.642863 3598 external_mini_cluster.cc:1428] Setting key f501e597780ce2d816fed930ed92952f
I20260621 05:10:22.643023 4763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:10:22.631122 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.131.129:45875 (local address 127.3.131.190:43199)
0621 05:10:22.631221 (+ 99us) server_negotiation.cc:206] Beginning negotiation
0621 05:10:22.631224 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0621 05:10:22.631421 (+ 197us) server_negotiation.cc:407] Connection header received
0621 05:10:22.632183 (+ 762us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:10:22.632186 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:10:22.632203 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:10:22.632280 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:10:22.633307 (+ 1027us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:10:22.634186 (+ 879us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:10:22.635992 (+ 1806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:10:22.636385 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:10:22.636471 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:10:22.639491 (+ 3020us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:10:22.639492 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:10:22.639492 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:10:22.639504 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:10:22.641128 (+ 1624us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:10:22.641629 (+ 501us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:10:22.641631 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:10:22.641631 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:10:22.641695 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:10:22.642153 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:10:22.642154 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:10:22.642154 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:10:22.642275 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:10:22.642329 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0621 05:10:22.642775 (+ 446us) server_negotiation.cc:299] Negotiation successful
0621 05:10:22.642895 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260621 05:10:22.643285 4809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:10:22.631034 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.131.190:43199 (local address 127.3.131.129:45875)
0621 05:10:22.631292 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0621 05:10:22.631332 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0621 05:10:22.632023 (+ 691us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:10:22.632405 (+ 382us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:10:22.632410 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:10:22.632829 (+ 419us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:10:22.633179 (+ 350us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:10:22.633187 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:10:22.634299 (+ 1112us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:10:22.634300 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:10:22.635892 (+ 1592us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:10:22.635895 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:10:22.636477 (+ 582us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:10:22.636478 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:10:22.636692 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:10:22.637334 (+ 642us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 05:10:22.637344 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:10:22.639388 (+ 2044us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 05:10:22.641232 (+ 1844us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:10:22.641239 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:10:22.641241 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:10:22.641481 (+ 240us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:10:22.641788 (+ 307us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:10:22.641789 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:10:22.641789 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:10:22.642030 (+ 241us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:10:22.642372 (+ 342us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 05:10:22.642378 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 05:10:22.642536 (+ 158us) client_negotiation.cc:769] Sending connection context
0621 05:10:22.642707 (+ 171us) client_negotiation.cc:240] Negotiation successful
0621 05:10:22.642900 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":204,"thread_start_us":88,"threads_started":1}
I20260621 05:10:22.644960 4916 heartbeater.cc:347] Connected to a master server at 127.3.131.190:43199
I20260621 05:10:22.645126 4916 heartbeater.cc:464] Registering TS with master...
I20260621 05:10:22.645375 4916 heartbeater.cc:511] Master 127.3.131.190:43199 requested a full tablet report, sending...
I20260621 05:10:22.646006 4712 ts_manager.cc:194] Registered new tserver with Master: 4f7fee4354ab4ed5b1bfae3ec9f66d4a (127.3.131.129:41195)
I20260621 05:10:22.646682 4712 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.131.129@KRBTEST.COM'} at 127.3.131.129:45875
I20260621 05:10:22.652376 4763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:10:22.644968 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.131.129:58909 (local address 127.3.131.190:43199)
0621 05:10:22.645014 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0621 05:10:22.645016 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:10:22.645169 (+ 153us) server_negotiation.cc:407] Connection header received
0621 05:10:22.645264 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:10:22.645265 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:10:22.645276 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:10:22.645338 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:10:22.645782 (+ 444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:10:22.646772 (+ 990us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:10:22.648168 (+ 1396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:10:22.648357 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:10:22.648457 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:10:22.649661 (+ 1204us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:10:22.649662 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:10:22.649662 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:10:22.649669 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:10:22.651103 (+ 1434us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:10:22.651569 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:10:22.651570 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:10:22.651571 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:10:22.651663 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:10:22.651896 (+ 233us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:10:22.651897 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:10:22.651897 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:10:22.652003 (+ 106us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:10:22.652055 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0621 05:10:22.652240 (+ 185us) server_negotiation.cc:299] Negotiation successful
0621 05:10:22.652311 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":129,"server-negotiator.queue_time_us":23}
I20260621 05:10:22.652465 4923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:10:22.644933 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.131.190:43199 (local address 127.3.131.129:58909)
0621 05:10:22.645083 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0621 05:10:22.645103 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0621 05:10:22.645196 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:10:22.645344 (+ 148us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:10:22.645345 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:10:22.645621 (+ 276us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:10:22.645715 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:10:22.645718 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:10:22.646965 (+ 1247us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:10:22.646966 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:10:22.648068 (+ 1102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:10:22.648072 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:10:22.648444 (+ 372us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:10:22.648445 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:10:22.648564 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:10:22.649032 (+ 468us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 05:10:22.649039 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:10:22.649553 (+ 514us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 05:10:22.651221 (+ 1668us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:10:22.651223 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:10:22.651223 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:10:22.651474 (+ 251us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:10:22.651745 (+ 271us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:10:22.651746 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:10:22.651746 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:10:22.651825 (+ 79us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:10:22.652076 (+ 251us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 05:10:22.652077 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 05:10:22.652135 (+ 58us) client_negotiation.cc:769] Sending connection context
0621 05:10:22.652225 (+ 90us) client_negotiation.cc:240] Negotiation successful
0621 05:10:22.652339 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":76,"threads_started":1}
WARNING: no policy specified for kudu/127.3.131.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.131.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.131.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782018616648578-3598-0/krb5kdc/kudu_127.3.131.130.keytab.
Entry for principal kudu/127.3.131.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782018616648578-3598-0/krb5kdc/kudu_127.3.131.130.keytab.
I20260621 05:10:22.675069 3598 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWZOx2_/build/release/bin/kudu
/tmp/dist-test-taskWZOx2_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782018616648578-3598-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782018616648578-3598-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782018616648578-3598-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782018616648578-3598-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.131.130:0
--local_ip_for_outbound_sockets=127.3.131.130
--webserver_interface=127.3.131.130
--webserver_port=0
--tserver_master_addrs=127.3.131.190:43199
--keytab_file=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782018616648578-3598-0/krb5kdc/kudu_127.3.131.130.keytab
--principal=kudu/127.3.131.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.131.148:41101
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782018616648578-3598-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782018616648578-3598-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782018616648578-3598-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 05:10:22.760011 4925 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 05:10:22.760375 4925 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 05:10:22.760406 4925 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 05:10:22.761801 4925 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 05:10:22.761967 4925 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 05:10:22.762012 4925 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260621 05:10:22.762033 4925 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.131.130
I20260621 05:10:22.763799 4925 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.131.148:41101
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782018616648578-3598-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782018616648578-3598-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782018616648578-3598-0/krb5kdc/kudu_127.3.131.130.keytab
--principal=kudu/127.3.131.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.131.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782018616648578-3598-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.131.130
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.3.131.190:43199
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4925
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.131.130
--log_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782018616648578-3598-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 21 Jun 2026 04:16:16 UTC on ccycloud.quasar-xycaha.root.comops.site
build id 4457881
I20260621 05:10:22.764030 4925 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 05:10:22.764218 4925 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 05:10:22.770047 4931 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:10:22.770051 4933 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:10:22.770051 4930 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:10:22.770319 4925 server_base.cc:1034] running on GCE node
I20260621 05:10:22.770514 4925 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 05:10:22.770744 4925 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 05:10:22.771968 4925 hybrid_clock.cc:648] HybridClock initialized: now 1782018622771950 us; error 30 us; skew 500 ppm
Jun 21 05:10:22 dist-test-slave-vrq1 krb5kdc[4668](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782018622, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.131.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 05:10:22.775215 4925 init.cc:377] Logged in from keytab as kudu/127.3.131.130@KRBTEST.COM (short username kudu)
I20260621 05:10:22.775811 4925 webserver.cc:460] Webserver started at http://127.3.131.130:39107/ using document root <none> and password file <none>
I20260621 05:10:22.776074 4925 fs_manager.cc:359] Metadata directory not provided
I20260621 05:10:22.776229 4925 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 05:10:22.776410 4925 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 05:10:22.777858 4925 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782018616648578-3598-0/minicluster-data/ts-1/data/instance:
uuid: "98aae82ed2bd426fb5bd4fd0a69ede6f"
format_stamp: "Formatted at 2026-06-21 05:10:22 on dist-test-slave-vrq1"
server_key: "a66308e77512b60f1b0bacc19e601675"
server_key_iv: "0c570a50d0e936ee2e7b02987a155088"
server_key_version: "encryptionkey@0"
I20260621 05:10:22.778276 4925 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782018616648578-3598-0/minicluster-data/ts-1/wal/instance:
uuid: "98aae82ed2bd426fb5bd4fd0a69ede6f"
format_stamp: "Formatted at 2026-06-21 05:10:22 on dist-test-slave-vrq1"
server_key: "a66308e77512b60f1b0bacc19e601675"
server_key_iv: "0c570a50d0e936ee2e7b02987a155088"
server_key_version: "encryptionkey@0"
I20260621 05:10:22.779862 4925 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.004s sys 0.000s
I20260621 05:10:22.780987 4940 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 05:10:22.781258 4925 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260621 05:10:22.781402 4925 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782018616648578-3598-0/minicluster-data/ts-1/data,/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782018616648578-3598-0/minicluster-data/ts-1/wal
uuid: "98aae82ed2bd426fb5bd4fd0a69ede6f"
format_stamp: "Formatted at 2026-06-21 05:10:22 on dist-test-slave-vrq1"
server_key: "a66308e77512b60f1b0bacc19e601675"
server_key_iv: "0c570a50d0e936ee2e7b02987a155088"
server_key_version: "encryptionkey@0"
I20260621 05:10:22.781489 4925 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782018616648578-3598-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782018616648578-3598-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782018616648578-3598-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 05:10:22.794781 4925 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 05:10:22.797361 4925 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 05:10:22.797549 4925 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 05:10:22.798180 4925 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 05:10:22.798254 4925 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 05:10:22.798288 4925 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 05:10:22.798309 4925 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 05:10:22.806124 4925 rpc_server.cc:307] RPC server started. Bound to: 127.3.131.130:44133
I20260621 05:10:22.806164 5055 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.131.130:44133 every 8 connection(s)
Jun 21 05:10:22 dist-test-slave-vrq1 krb5kdc[4668](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782018622, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.131.130@KRBTEST.COM for kudu/127.3.131.190@KRBTEST.COM
I20260621 05:10:22.806702 4925 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782018616648578-3598-0/minicluster-data/ts-1/data/info.pb
I20260621 05:10:22.810451 4763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:10:22.798828 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.131.130:34033 (local address 127.3.131.190:43199)
0621 05:10:22.798893 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0621 05:10:22.798895 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:10:22.799129 (+ 234us) server_negotiation.cc:407] Connection header received
0621 05:10:22.799863 (+ 734us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:10:22.799864 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:10:22.799878 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:10:22.799930 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:10:22.800887 (+ 957us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:10:22.801807 (+ 920us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:10:22.803572 (+ 1765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:10:22.803898 (+ 326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:10:22.804027 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:10:22.807391 (+ 3364us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:10:22.807393 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:10:22.807393 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:10:22.807401 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:10:22.809010 (+ 1609us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:10:22.809480 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:10:22.809481 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:10:22.809481 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:10:22.809539 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:10:22.809831 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:10:22.809832 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:10:22.809832 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:10:22.809965 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:10:22.810026 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0621 05:10:22.810306 (+ 280us) server_negotiation.cc:299] Negotiation successful
0621 05:10:22.810385 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260621 05:10:22.810604 4949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:10:22.798826 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.131.190:43199 (local address 127.3.131.130:34033)
0621 05:10:22.799007 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0621 05:10:22.799045 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0621 05:10:22.799729 (+ 684us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:10:22.800007 (+ 278us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:10:22.800012 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:10:22.800423 (+ 411us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:10:22.800795 (+ 372us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:10:22.800804 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:10:22.801910 (+ 1106us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:10:22.801911 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:10:22.803439 (+ 1528us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:10:22.803442 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:10:22.804040 (+ 598us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:10:22.804041 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:10:22.804193 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:10:22.804939 (+ 746us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 05:10:22.804954 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:10:22.807239 (+ 2285us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 05:10:22.809110 (+ 1871us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:10:22.809118 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:10:22.809120 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:10:22.809389 (+ 269us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:10:22.809603 (+ 214us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:10:22.809603 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:10:22.809604 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:10:22.809730 (+ 126us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:10:22.810029 (+ 299us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 05:10:22.810033 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 05:10:22.810177 (+ 144us) client_negotiation.cc:769] Sending connection context
0621 05:10:22.810259 (+ 82us) client_negotiation.cc:240] Negotiation successful
0621 05:10:22.810395 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":102,"threads_started":1}
I20260621 05:10:22.811060 3598 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWZOx2_/build/release/bin/kudu as pid 4925
I20260621 05:10:22.811241 3598 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782018616648578-3598-0/minicluster-data/ts-1/wal/instance
I20260621 05:10:22.811252 5056 heartbeater.cc:347] Connected to a master server at 127.3.131.190:43199
I20260621 05:10:22.811331 5056 heartbeater.cc:464] Registering TS with master...
I20260621 05:10:22.811451 3598 external_mini_cluster.cc:1428] Setting key 8c4922cd5f389c25312186ebb44a3c5f
I20260621 05:10:22.811493 5056 heartbeater.cc:511] Master 127.3.131.190:43199 requested a full tablet report, sending...
I20260621 05:10:22.811852 4712 ts_manager.cc:194] Registered new tserver with Master: 98aae82ed2bd426fb5bd4fd0a69ede6f (127.3.131.130:44133)
I20260621 05:10:22.812384 4712 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.131.130@KRBTEST.COM'} at 127.3.131.130:34033
I20260621 05:10:22.819348 4763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:10:22.812475 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.131.130:42107 (local address 127.3.131.190:43199)
0621 05:10:22.812514 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0621 05:10:22.812516 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:10:22.812619 (+ 103us) server_negotiation.cc:407] Connection header received
0621 05:10:22.812720 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:10:22.812721 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:10:22.812731 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:10:22.812772 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:10:22.813173 (+ 401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:10:22.813993 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:10:22.815047 (+ 1054us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:10:22.815347 (+ 300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:10:22.815405 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:10:22.816529 (+ 1124us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:10:22.816530 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:10:22.816530 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:10:22.816536 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:10:22.817977 (+ 1441us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:10:22.818377 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:10:22.818378 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:10:22.818378 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:10:22.818431 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:10:22.818685 (+ 254us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:10:22.818686 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:10:22.818687 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:10:22.818846 (+ 159us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:10:22.818912 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0621 05:10:22.819093 (+ 181us) server_negotiation.cc:299] Negotiation successful
0621 05:10:22.819281 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260621 05:10:22.819342 5062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:10:22.812414 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.131.190:43199 (local address 127.3.131.130:42107)
0621 05:10:22.812559 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0621 05:10:22.812569 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0621 05:10:22.812660 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:10:22.812816 (+ 156us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:10:22.812817 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:10:22.813022 (+ 205us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:10:22.813082 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:10:22.813084 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:10:22.814097 (+ 1013us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:10:22.814099 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:10:22.814963 (+ 864us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:10:22.814966 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:10:22.815421 (+ 455us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:10:22.815422 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:10:22.815509 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:10:22.815985 (+ 476us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 05:10:22.815991 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:10:22.816437 (+ 446us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 05:10:22.818063 (+ 1626us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:10:22.818064 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:10:22.818065 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:10:22.818299 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:10:22.818497 (+ 198us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:10:22.818498 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:10:22.818498 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:10:22.818585 (+ 87us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:10:22.818927 (+ 342us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 05:10:22.818928 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 05:10:22.818986 (+ 58us) client_negotiation.cc:769] Sending connection context
0621 05:10:22.819066 (+ 80us) client_negotiation.cc:240] Negotiation successful
0621 05:10:22.819214 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":75,"threads_started":1}
WARNING: no policy specified for kudu/127.3.131.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.131.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.131.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782018616648578-3598-0/krb5kdc/kudu_127.3.131.131.keytab.
Entry for principal kudu/127.3.131.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782018616648578-3598-0/krb5kdc/kudu_127.3.131.131.keytab.
I20260621 05:10:22.842155 3598 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWZOx2_/build/release/bin/kudu
/tmp/dist-test-taskWZOx2_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782018616648578-3598-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782018616648578-3598-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782018616648578-3598-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782018616648578-3598-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.131.131:0
--local_ip_for_outbound_sockets=127.3.131.131
--webserver_interface=127.3.131.131
--webserver_port=0
--tserver_master_addrs=127.3.131.190:43199
--keytab_file=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782018616648578-3598-0/krb5kdc/kudu_127.3.131.131.keytab
--principal=kudu/127.3.131.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.131.148:41101
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782018616648578-3598-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782018616648578-3598-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782018616648578-3598-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 05:10:22.927126 5065 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 05:10:22.927506 5065 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 05:10:22.927569 5065 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 05:10:22.928812 5065 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 05:10:22.928854 5065 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 05:10:22.928876 5065 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260621 05:10:22.928894 5065 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.131.131
I20260621 05:10:22.930472 5065 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.131.148:41101
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782018616648578-3598-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782018616648578-3598-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782018616648578-3598-0/krb5kdc/kudu_127.3.131.131.keytab
--principal=kudu/127.3.131.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.131.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782018616648578-3598-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.131.131
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.3.131.190:43199
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5065
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.131.131
--log_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782018616648578-3598-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 21 Jun 2026 04:16:16 UTC on ccycloud.quasar-xycaha.root.comops.site
build id 4457881
I20260621 05:10:22.930722 5065 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 05:10:22.930958 5065 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 05:10:22.935935 5073 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:10:22.935961 5071 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:10:22.935968 5070 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:10:22.936267 5065 server_base.cc:1034] running on GCE node
I20260621 05:10:22.936475 5065 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 05:10:22.936721 5065 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 05:10:22.937889 5065 hybrid_clock.cc:648] HybridClock initialized: now 1782018622937889 us; error 41 us; skew 500 ppm
Jun 21 05:10:22 dist-test-slave-vrq1 krb5kdc[4668](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782018622, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.131.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 05:10:22.940408 5065 init.cc:377] Logged in from keytab as kudu/127.3.131.131@KRBTEST.COM (short username kudu)
I20260621 05:10:22.940877 5065 webserver.cc:460] Webserver started at http://127.3.131.131:41245/ using document root <none> and password file <none>
I20260621 05:10:22.941114 5065 fs_manager.cc:359] Metadata directory not provided
I20260621 05:10:22.941192 5065 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 05:10:22.941347 5065 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 05:10:22.942483 5065 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782018616648578-3598-0/minicluster-data/ts-2/data/instance:
uuid: "b9643924b7a2495cbae396cd2b2523b4"
format_stamp: "Formatted at 2026-06-21 05:10:22 on dist-test-slave-vrq1"
server_key: "82ad6481cb117bc33425fb43a7321b68"
server_key_iv: "354054804a04f93cbe0fb169602caa09"
server_key_version: "encryptionkey@0"
I20260621 05:10:22.942888 5065 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782018616648578-3598-0/minicluster-data/ts-2/wal/instance:
uuid: "b9643924b7a2495cbae396cd2b2523b4"
format_stamp: "Formatted at 2026-06-21 05:10:22 on dist-test-slave-vrq1"
server_key: "82ad6481cb117bc33425fb43a7321b68"
server_key_iv: "354054804a04f93cbe0fb169602caa09"
server_key_version: "encryptionkey@0"
I20260621 05:10:22.944497 5065 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260621 05:10:22.945345 5080 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 05:10:22.945576 5065 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260621 05:10:22.945673 5065 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782018616648578-3598-0/minicluster-data/ts-2/data,/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782018616648578-3598-0/minicluster-data/ts-2/wal
uuid: "b9643924b7a2495cbae396cd2b2523b4"
format_stamp: "Formatted at 2026-06-21 05:10:22 on dist-test-slave-vrq1"
server_key: "82ad6481cb117bc33425fb43a7321b68"
server_key_iv: "354054804a04f93cbe0fb169602caa09"
server_key_version: "encryptionkey@0"
I20260621 05:10:22.945760 5065 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782018616648578-3598-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782018616648578-3598-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782018616648578-3598-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 05:10:22.960265 5065 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 05:10:22.962692 5065 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 05:10:22.962911 5065 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 05:10:22.963572 5065 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 05:10:22.963634 5065 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 05:10:22.963675 5065 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 05:10:22.963707 5065 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 05:10:22.970840 5065 rpc_server.cc:307] RPC server started. Bound to: 127.3.131.131:45593
I20260621 05:10:22.970934 5195 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.131.131:45593 every 8 connection(s)
Jun 21 05:10:22 dist-test-slave-vrq1 krb5kdc[4668](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782018622, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.131.131@KRBTEST.COM for kudu/127.3.131.190@KRBTEST.COM
I20260621 05:10:22.971292 5065 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782018616648578-3598-0/minicluster-data/ts-2/data/info.pb
I20260621 05:10:22.974944 4763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:10:22.963974 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.131.131:38057 (local address 127.3.131.190:43199)
0621 05:10:22.964044 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0621 05:10:22.964047 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0621 05:10:22.964278 (+ 231us) server_negotiation.cc:407] Connection header received
0621 05:10:22.964825 (+ 547us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:10:22.964826 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:10:22.964847 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:10:22.964902 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:10:22.965707 (+ 805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:10:22.966237 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:10:22.968049 (+ 1812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:10:22.968374 (+ 325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:10:22.968482 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:10:22.971736 (+ 3254us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:10:22.971738 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:10:22.971738 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:10:22.971749 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:10:22.973342 (+ 1593us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:10:22.973887 (+ 545us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:10:22.973888 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:10:22.973888 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:10:22.973950 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:10:22.974277 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:10:22.974278 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:10:22.974278 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:10:22.974413 (+ 135us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:10:22.974483 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0621 05:10:22.974796 (+ 313us) server_negotiation.cc:299] Negotiation successful
0621 05:10:22.974876 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260621 05:10:22.975108 5089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:10:22.963971 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.131.190:43199 (local address 127.3.131.131:38057)
0621 05:10:22.964146 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0621 05:10:22.964187 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0621 05:10:22.964712 (+ 525us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:10:22.964982 (+ 270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:10:22.964986 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:10:22.965346 (+ 360us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:10:22.965566 (+ 220us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:10:22.965572 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:10:22.966347 (+ 775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:10:22.966349 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:10:22.967918 (+ 1569us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:10:22.967924 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:10:22.968504 (+ 580us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:10:22.968505 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:10:22.968641 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:10:22.969757 (+ 1116us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 05:10:22.969768 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:10:22.971588 (+ 1820us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 05:10:22.973475 (+ 1887us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:10:22.973484 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:10:22.973486 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:10:22.973774 (+ 288us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:10:22.974026 (+ 252us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:10:22.974027 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:10:22.974028 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:10:22.974174 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:10:22.974488 (+ 314us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 05:10:22.974493 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 05:10:22.974627 (+ 134us) client_negotiation.cc:769] Sending connection context
0621 05:10:22.974732 (+ 105us) client_negotiation.cc:240] Negotiation successful
0621 05:10:22.974888 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":91,"threads_started":1}
I20260621 05:10:22.975777 5196 heartbeater.cc:347] Connected to a master server at 127.3.131.190:43199
I20260621 05:10:22.975872 5196 heartbeater.cc:464] Registering TS with master...
I20260621 05:10:22.976073 5196 heartbeater.cc:511] Master 127.3.131.190:43199 requested a full tablet report, sending...
I20260621 05:10:22.976547 4713 ts_manager.cc:194] Registered new tserver with Master: b9643924b7a2495cbae396cd2b2523b4 (127.3.131.131:45593)
I20260621 05:10:22.977020 4713 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.131.131@KRBTEST.COM'} at 127.3.131.131:38057
I20260621 05:10:22.978029 3598 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWZOx2_/build/release/bin/kudu as pid 5065
I20260621 05:10:22.978099 3598 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782018616648578-3598-0/minicluster-data/ts-2/wal/instance
I20260621 05:10:22.978281 3598 external_mini_cluster.cc:1428] Setting key a8874eabe13b51e91e0fd1698d183142
I20260621 05:10:22.979025 3598 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260621 05:10:22.983271 4763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:10:22.977016 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.131.131:41455 (local address 127.3.131.190:43199)
0621 05:10:22.977061 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0621 05:10:22.977063 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:10:22.977156 (+ 93us) server_negotiation.cc:407] Connection header received
0621 05:10:22.977224 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:10:22.977224 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:10:22.977235 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:10:22.977279 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:10:22.977686 (+ 407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:10:22.978329 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:10:22.979077 (+ 748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:10:22.979380 (+ 303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:10:22.979457 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:10:22.980495 (+ 1038us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:10:22.980496 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:10:22.980497 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:10:22.980504 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:10:22.982107 (+ 1603us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:10:22.982449 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:10:22.982450 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:10:22.982450 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:10:22.982506 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:10:22.982719 (+ 213us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:10:22.982720 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:10:22.982720 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:10:22.982834 (+ 114us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:10:22.982904 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0621 05:10:22.983060 (+ 156us) server_negotiation.cc:299] Negotiation successful
0621 05:10:22.983145 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260621 05:10:22.983289 5202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:10:22.976954 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.131.190:43199 (local address 127.3.131.131:41455)
0621 05:10:22.977075 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0621 05:10:22.977088 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0621 05:10:22.977165 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:10:22.977281 (+ 116us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:10:22.977282 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:10:22.977511 (+ 229us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:10:22.977579 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:10:22.977581 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:10:22.978427 (+ 846us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:10:22.978428 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:10:22.978985 (+ 557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:10:22.978987 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:10:22.979455 (+ 468us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:10:22.979456 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:10:22.979543 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:10:22.980081 (+ 538us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 05:10:22.980086 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:10:22.980385 (+ 299us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 05:10:22.982185 (+ 1800us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:10:22.982186 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:10:22.982187 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:10:22.982363 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:10:22.982577 (+ 214us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:10:22.982578 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:10:22.982579 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:10:22.982646 (+ 67us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:10:22.982906 (+ 260us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 05:10:22.982907 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 05:10:22.982951 (+ 44us) client_negotiation.cc:769] Sending connection context
0621 05:10:22.983036 (+ 85us) client_negotiation.cc:240] Negotiation successful
0621 05:10:22.983147 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":94,"thread_start_us":68,"threads_started":1}
Jun 21 05:10:22 dist-test-slave-vrq1 krb5kdc[4668](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782018622, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 21 05:10:22 dist-test-slave-vrq1 krb5kdc[4668](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782018622, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.3.131.190@KRBTEST.COM
I20260621 05:10:23.003686 4763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:10:22.995999 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46762 (local address 127.3.131.190:43199)
0621 05:10:22.996073 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0621 05:10:22.996075 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:10:22.996086 (+ 11us) server_negotiation.cc:407] Connection header received
0621 05:10:22.996123 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:10:22.996124 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:10:22.996133 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:10:22.996189 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:10:22.996846 (+ 657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:10:22.997347 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:10:22.998118 (+ 771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:10:22.998324 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:10:22.998396 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:10:23.000905 (+ 2509us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:10:23.000906 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:10:23.000906 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:10:23.000914 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:10:23.002409 (+ 1495us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:10:23.002803 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:10:23.002804 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:10:23.002804 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:10:23.002859 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:10:23.003044 (+ 185us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:10:23.003045 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:10:23.003045 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:10:23.003286 (+ 241us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:10:23.003375 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0621 05:10:23.003513 (+ 138us) server_negotiation.cc:299] Negotiation successful
0621 05:10:23.003578 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260621 05:10:23.005494 4713 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46762:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260621 05:10:23.007134 4713 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260621 05:10:23.015444 5215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:10:23.010069 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.131.131:45593 (local address 127.0.0.1:54090)
0621 05:10:23.010342 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0621 05:10:23.010393 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0621 05:10:23.010463 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:10:23.011133 (+ 670us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:10:23.011133 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:10:23.011135 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 05:10:23.011278 (+ 143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:10:23.011280 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:10:23.012738 (+ 1458us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:10:23.012739 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:10:23.014112 (+ 1373us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:10:23.014115 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:10:23.015160 (+ 1045us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:10:23.015161 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:10:23.015265 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:10:23.015270 (+ 5us) client_negotiation.cc:769] Sending connection context
0621 05:10:23.015304 (+ 34us) client_negotiation.cc:240] Negotiation successful
0621 05:10:23.015349 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":234,"mutex_wait_us":3,"thread_start_us":58,"threads_started":1}
I20260621 05:10:23.015583 5218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:10:23.010535 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51878 (local address 127.3.131.129:41195)
0621 05:10:23.010731 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0621 05:10:23.010737 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 05:10:23.010809 (+ 72us) server_negotiation.cc:407] Connection header received
0621 05:10:23.010892 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:10:23.010894 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:10:23.010909 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:10:23.011004 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 05:10:23.011274 (+ 270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:10:23.012141 (+ 867us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:10:23.013946 (+ 1805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:10:23.014902 (+ 956us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:10:23.014984 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:10:23.015058 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0621 05:10:23.015401 (+ 343us) server_negotiation.cc:299] Negotiation successful
0621 05:10:23.015453 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":163,"thread_start_us":130,"threads_started":1}
I20260621 05:10:23.015693 5216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:10:23.010303 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.131.129:41195 (local address 127.0.0.1:51878)
0621 05:10:23.010757 (+ 454us) negotiation.cc:107] Waiting for socket to connect
0621 05:10:23.010766 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0621 05:10:23.010820 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:10:23.011081 (+ 261us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:10:23.011082 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:10:23.011084 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 05:10:23.011159 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:10:23.011161 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:10:23.012234 (+ 1073us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:10:23.012235 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:10:23.013833 (+ 1598us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:10:23.013837 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:10:23.015004 (+ 1167us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:10:23.015005 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:10:23.015073 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:10:23.015085 (+ 12us) client_negotiation.cc:769] Sending connection context
0621 05:10:23.015184 (+ 99us) client_negotiation.cc:240] Negotiation successful
0621 05:10:23.015278 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":437,"mutex_wait_us":8,"thread_start_us":86,"threads_started":1}
I20260621 05:10:23.016085 5214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:10:23.010179 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54090 (local address 127.3.131.131:45593)
0621 05:10:23.010577 (+ 398us) server_negotiation.cc:206] Beginning negotiation
0621 05:10:23.010582 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 05:10:23.010593 (+ 11us) server_negotiation.cc:407] Connection header received
0621 05:10:23.010640 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:10:23.010642 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:10:23.010659 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:10:23.011369 (+ 710us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 05:10:23.011429 (+ 60us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:10:23.012652 (+ 1223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:10:23.014223 (+ 1571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:10:23.015061 (+ 838us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:10:23.015838 (+ 777us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:10:23.015915 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0621 05:10:23.015942 (+ 27us) server_negotiation.cc:299] Negotiation successful
0621 05:10:23.015985 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":360,"thread_start_us":80,"threads_started":1}
I20260621 05:10:23.016506 4850 tablet_service.cc:1467] Processing CreateTablet for tablet be5721a5e3b5497f95d3d5f7b99ef2c0 (DEFAULT_TABLE table=test-table [id=40f6a3e9f12d4bd8bac69db474d0f59e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 05:10:23.016671 5213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:10:23.010139 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.131.130:44133 (local address 127.0.0.1:58808)
0621 05:10:23.010347 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0621 05:10:23.010395 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0621 05:10:23.010463 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:10:23.011049 (+ 586us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:10:23.011052 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:10:23.011054 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 05:10:23.011133 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:10:23.011135 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:10:23.012653 (+ 1518us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:10:23.012654 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:10:23.013839 (+ 1185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:10:23.013841 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:10:23.016400 (+ 2559us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:10:23.016401 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:10:23.016482 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:10:23.016486 (+ 4us) client_negotiation.cc:769] Sending connection context
0621 05:10:23.016548 (+ 62us) client_negotiation.cc:240] Negotiation successful
0621 05:10:23.016592 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":188,"mutex_wait_us":28,"thread_start_us":143,"threads_started":1}
I20260621 05:10:23.016716 5217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:10:23.010376 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58808 (local address 127.3.131.130:44133)
0621 05:10:23.010879 (+ 503us) server_negotiation.cc:206] Beginning negotiation
0621 05:10:23.010884 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 05:10:23.010893 (+ 9us) server_negotiation.cc:407] Connection header received
0621 05:10:23.010934 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:10:23.010936 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:10:23.010951 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:10:23.011034 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 05:10:23.011276 (+ 242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:10:23.012564 (+ 1288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:10:23.015103 (+ 2539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:10:23.016322 (+ 1219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:10:23.016401 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:10:23.016460 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0621 05:10:23.016563 (+ 103us) server_negotiation.cc:299] Negotiation successful
0621 05:10:23.016602 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":465,"thread_start_us":76,"threads_started":1}
I20260621 05:10:23.016696 5130 tablet_service.cc:1467] Processing CreateTablet for tablet be5721a5e3b5497f95d3d5f7b99ef2c0 (DEFAULT_TABLE table=test-table [id=40f6a3e9f12d4bd8bac69db474d0f59e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 05:10:23.016916 4850 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet be5721a5e3b5497f95d3d5f7b99ef2c0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 05:10:23.016990 5130 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet be5721a5e3b5497f95d3d5f7b99ef2c0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 05:10:23.017292 4990 tablet_service.cc:1467] Processing CreateTablet for tablet be5721a5e3b5497f95d3d5f7b99ef2c0 (DEFAULT_TABLE table=test-table [id=40f6a3e9f12d4bd8bac69db474d0f59e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 05:10:23.017555 4990 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet be5721a5e3b5497f95d3d5f7b99ef2c0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 05:10:23.020417 5219 tablet_bootstrap.cc:492] T be5721a5e3b5497f95d3d5f7b99ef2c0 P b9643924b7a2495cbae396cd2b2523b4: Bootstrap starting.
I20260621 05:10:23.021075 5221 tablet_bootstrap.cc:492] T be5721a5e3b5497f95d3d5f7b99ef2c0 P 4f7fee4354ab4ed5b1bfae3ec9f66d4a: Bootstrap starting.
I20260621 05:10:23.021623 5219 tablet_bootstrap.cc:654] T be5721a5e3b5497f95d3d5f7b99ef2c0 P b9643924b7a2495cbae396cd2b2523b4: Neither blocks nor log segments found. Creating new log.
I20260621 05:10:23.021641 5221 tablet_bootstrap.cc:654] T be5721a5e3b5497f95d3d5f7b99ef2c0 P 4f7fee4354ab4ed5b1bfae3ec9f66d4a: Neither blocks nor log segments found. Creating new log.
I20260621 05:10:23.021965 5221 log.cc:824] T be5721a5e3b5497f95d3d5f7b99ef2c0 P 4f7fee4354ab4ed5b1bfae3ec9f66d4a: Log is configured to *not* fsync() on all Append() calls
I20260621 05:10:23.021972 5219 log.cc:824] T be5721a5e3b5497f95d3d5f7b99ef2c0 P b9643924b7a2495cbae396cd2b2523b4: Log is configured to *not* fsync() on all Append() calls
I20260621 05:10:23.022349 5220 tablet_bootstrap.cc:492] T be5721a5e3b5497f95d3d5f7b99ef2c0 P 98aae82ed2bd426fb5bd4fd0a69ede6f: Bootstrap starting.
I20260621 05:10:23.022655 5221 tablet_bootstrap.cc:492] T be5721a5e3b5497f95d3d5f7b99ef2c0 P 4f7fee4354ab4ed5b1bfae3ec9f66d4a: No bootstrap required, opened a new log
I20260621 05:10:23.022739 5219 tablet_bootstrap.cc:492] T be5721a5e3b5497f95d3d5f7b99ef2c0 P b9643924b7a2495cbae396cd2b2523b4: No bootstrap required, opened a new log
I20260621 05:10:23.022733 5221 ts_tablet_manager.cc:1397] T be5721a5e3b5497f95d3d5f7b99ef2c0 P 4f7fee4354ab4ed5b1bfae3ec9f66d4a: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260621 05:10:23.022830 5219 ts_tablet_manager.cc:1397] T be5721a5e3b5497f95d3d5f7b99ef2c0 P b9643924b7a2495cbae396cd2b2523b4: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260621 05:10:23.023880 5220 tablet_bootstrap.cc:654] T be5721a5e3b5497f95d3d5f7b99ef2c0 P 98aae82ed2bd426fb5bd4fd0a69ede6f: Neither blocks nor log segments found. Creating new log.
I20260621 05:10:23.024250 5220 log.cc:824] T be5721a5e3b5497f95d3d5f7b99ef2c0 P 98aae82ed2bd426fb5bd4fd0a69ede6f: Log is configured to *not* fsync() on all Append() calls
I20260621 05:10:23.024549 5221 raft_consensus.cc:348] T be5721a5e3b5497f95d3d5f7b99ef2c0 P 4f7fee4354ab4ed5b1bfae3ec9f66d4a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9643924b7a2495cbae396cd2b2523b4" member_type: VOTER last_known_addr { host: "127.3.131.131" port: 45593 } } peers { permanent_uuid: "4f7fee4354ab4ed5b1bfae3ec9f66d4a" member_type: VOTER last_known_addr { host: "127.3.131.129" port: 41195 } } peers { permanent_uuid: "98aae82ed2bd426fb5bd4fd0a69ede6f" member_type: VOTER last_known_addr { host: "127.3.131.130" port: 44133 } }
I20260621 05:10:23.024706 5221 raft_consensus.cc:374] T be5721a5e3b5497f95d3d5f7b99ef2c0 P 4f7fee4354ab4ed5b1bfae3ec9f66d4a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 05:10:23.024749 5221 raft_consensus.cc:729] T be5721a5e3b5497f95d3d5f7b99ef2c0 P 4f7fee4354ab4ed5b1bfae3ec9f66d4a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4f7fee4354ab4ed5b1bfae3ec9f66d4a, State: Initialized, Role: FOLLOWER
I20260621 05:10:23.024873 5221 consensus_queue.cc:260] T be5721a5e3b5497f95d3d5f7b99ef2c0 P 4f7fee4354ab4ed5b1bfae3ec9f66d4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9643924b7a2495cbae396cd2b2523b4" member_type: VOTER last_known_addr { host: "127.3.131.131" port: 45593 } } peers { permanent_uuid: "4f7fee4354ab4ed5b1bfae3ec9f66d4a" member_type: VOTER last_known_addr { host: "127.3.131.129" port: 41195 } } peers { permanent_uuid: "98aae82ed2bd426fb5bd4fd0a69ede6f" member_type: VOTER last_known_addr { host: "127.3.131.130" port: 44133 } }
I20260621 05:10:23.024993 5219 raft_consensus.cc:348] T be5721a5e3b5497f95d3d5f7b99ef2c0 P b9643924b7a2495cbae396cd2b2523b4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9643924b7a2495cbae396cd2b2523b4" member_type: VOTER last_known_addr { host: "127.3.131.131" port: 45593 } } peers { permanent_uuid: "4f7fee4354ab4ed5b1bfae3ec9f66d4a" member_type: VOTER last_known_addr { host: "127.3.131.129" port: 41195 } } peers { permanent_uuid: "98aae82ed2bd426fb5bd4fd0a69ede6f" member_type: VOTER last_known_addr { host: "127.3.131.130" port: 44133 } }
I20260621 05:10:23.025133 5221 ts_tablet_manager.cc:1428] T be5721a5e3b5497f95d3d5f7b99ef2c0 P 4f7fee4354ab4ed5b1bfae3ec9f66d4a: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20260621 05:10:23.025218 4916 heartbeater.cc:503] Master 127.3.131.190:43199 was elected leader, sending a full tablet report...
I20260621 05:10:23.025449 5219 raft_consensus.cc:374] T be5721a5e3b5497f95d3d5f7b99ef2c0 P b9643924b7a2495cbae396cd2b2523b4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 05:10:23.025521 5219 raft_consensus.cc:729] T be5721a5e3b5497f95d3d5f7b99ef2c0 P b9643924b7a2495cbae396cd2b2523b4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b9643924b7a2495cbae396cd2b2523b4, State: Initialized, Role: FOLLOWER
I20260621 05:10:23.025677 5219 consensus_queue.cc:260] T be5721a5e3b5497f95d3d5f7b99ef2c0 P b9643924b7a2495cbae396cd2b2523b4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9643924b7a2495cbae396cd2b2523b4" member_type: VOTER last_known_addr { host: "127.3.131.131" port: 45593 } } peers { permanent_uuid: "4f7fee4354ab4ed5b1bfae3ec9f66d4a" member_type: VOTER last_known_addr { host: "127.3.131.129" port: 41195 } } peers { permanent_uuid: "98aae82ed2bd426fb5bd4fd0a69ede6f" member_type: VOTER last_known_addr { host: "127.3.131.130" port: 44133 } }
I20260621 05:10:23.026072 5220 tablet_bootstrap.cc:492] T be5721a5e3b5497f95d3d5f7b99ef2c0 P 98aae82ed2bd426fb5bd4fd0a69ede6f: No bootstrap required, opened a new log
I20260621 05:10:23.026122 5219 ts_tablet_manager.cc:1428] T be5721a5e3b5497f95d3d5f7b99ef2c0 P b9643924b7a2495cbae396cd2b2523b4: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260621 05:10:23.026152 5220 ts_tablet_manager.cc:1397] T be5721a5e3b5497f95d3d5f7b99ef2c0 P 98aae82ed2bd426fb5bd4fd0a69ede6f: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20260621 05:10:23.026203 5196 heartbeater.cc:503] Master 127.3.131.190:43199 was elected leader, sending a full tablet report...
I20260621 05:10:23.028017 5220 raft_consensus.cc:348] T be5721a5e3b5497f95d3d5f7b99ef2c0 P 98aae82ed2bd426fb5bd4fd0a69ede6f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9643924b7a2495cbae396cd2b2523b4" member_type: VOTER last_known_addr { host: "127.3.131.131" port: 45593 } } peers { permanent_uuid: "4f7fee4354ab4ed5b1bfae3ec9f66d4a" member_type: VOTER last_known_addr { host: "127.3.131.129" port: 41195 } } peers { permanent_uuid: "98aae82ed2bd426fb5bd4fd0a69ede6f" member_type: VOTER last_known_addr { host: "127.3.131.130" port: 44133 } }
I20260621 05:10:23.028178 5220 raft_consensus.cc:374] T be5721a5e3b5497f95d3d5f7b99ef2c0 P 98aae82ed2bd426fb5bd4fd0a69ede6f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 05:10:23.028224 5220 raft_consensus.cc:729] T be5721a5e3b5497f95d3d5f7b99ef2c0 P 98aae82ed2bd426fb5bd4fd0a69ede6f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98aae82ed2bd426fb5bd4fd0a69ede6f, State: Initialized, Role: FOLLOWER
I20260621 05:10:23.028370 5220 consensus_queue.cc:260] T be5721a5e3b5497f95d3d5f7b99ef2c0 P 98aae82ed2bd426fb5bd4fd0a69ede6f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9643924b7a2495cbae396cd2b2523b4" member_type: VOTER last_known_addr { host: "127.3.131.131" port: 45593 } } peers { permanent_uuid: "4f7fee4354ab4ed5b1bfae3ec9f66d4a" member_type: VOTER last_known_addr { host: "127.3.131.129" port: 41195 } } peers { permanent_uuid: "98aae82ed2bd426fb5bd4fd0a69ede6f" member_type: VOTER last_known_addr { host: "127.3.131.130" port: 44133 } }
I20260621 05:10:23.028695 5220 ts_tablet_manager.cc:1428] T be5721a5e3b5497f95d3d5f7b99ef2c0 P 98aae82ed2bd426fb5bd4fd0a69ede6f: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260621 05:10:23.028759 5056 heartbeater.cc:503] Master 127.3.131.190:43199 was elected leader, sending a full tablet report...
W20260621 05:10:23.057366 5057 tablet.cc:2369] T be5721a5e3b5497f95d3d5f7b99ef2c0 P 98aae82ed2bd426fb5bd4fd0a69ede6f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260621 05:10:23.062763 5227 raft_consensus.cc:482] T be5721a5e3b5497f95d3d5f7b99ef2c0 P 98aae82ed2bd426fb5bd4fd0a69ede6f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260621 05:10:23.062932 5227 raft_consensus.cc:504] T be5721a5e3b5497f95d3d5f7b99ef2c0 P 98aae82ed2bd426fb5bd4fd0a69ede6f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9643924b7a2495cbae396cd2b2523b4" member_type: VOTER last_known_addr { host: "127.3.131.131" port: 45593 } } peers { permanent_uuid: "4f7fee4354ab4ed5b1bfae3ec9f66d4a" member_type: VOTER last_known_addr { host: "127.3.131.129" port: 41195 } } peers { permanent_uuid: "98aae82ed2bd426fb5bd4fd0a69ede6f" member_type: VOTER last_known_addr { host: "127.3.131.130" port: 44133 } }
I20260621 05:10:23.063323 5227 leader_election.cc:288] T be5721a5e3b5497f95d3d5f7b99ef2c0 P 98aae82ed2bd426fb5bd4fd0a69ede6f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b9643924b7a2495cbae396cd2b2523b4 (127.3.131.131:45593), 4f7fee4354ab4ed5b1bfae3ec9f66d4a (127.3.131.129:41195)
I20260621 05:10:23.066599 5214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:10:23.063673 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.131.130:51863 (local address 127.3.131.131:45593)
0621 05:10:23.063743 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0621 05:10:23.063745 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:10:23.063753 (+ 8us) server_negotiation.cc:407] Connection header received
0621 05:10:23.063798 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:10:23.063799 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:10:23.063810 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:10:23.063881 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 05:10:23.064081 (+ 200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:10:23.064752 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:10:23.065784 (+ 1032us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:10:23.066209 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:10:23.066266 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:10:23.066274 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0621 05:10:23.066472 (+ 198us) server_negotiation.cc:299] Negotiation successful
0621 05:10:23.066519 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260621 05:10:23.066604 5218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:10:23.063839 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.131.130:41933 (local address 127.3.131.129:41195)
0621 05:10:23.063892 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0621 05:10:23.063894 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:10:23.064045 (+ 151us) server_negotiation.cc:407] Connection header received
0621 05:10:23.064145 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:10:23.064146 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:10:23.064153 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:10:23.064192 (+ 39us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 05:10:23.064394 (+ 202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:10:23.064919 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:10:23.065883 (+ 964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:10:23.066232 (+ 349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:10:23.066329 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:10:23.066337 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0621 05:10:23.066505 (+ 168us) server_negotiation.cc:299] Negotiation successful
0621 05:10:23.066543 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260621 05:10:23.066601 4949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:10:23.063550 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.131.131:45593 (local address 127.3.131.130:51863)
0621 05:10:23.063615 (+ 65us) negotiation.cc:107] Waiting for socket to connect
0621 05:10:23.063629 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0621 05:10:23.063683 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:10:23.063903 (+ 220us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:10:23.063904 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:10:23.063906 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 05:10:23.063980 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:10:23.063982 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:10:23.064847 (+ 865us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:10:23.064848 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:10:23.065684 (+ 836us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:10:23.065686 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:10:23.066289 (+ 603us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:10:23.066290 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:10:23.066390 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:10:23.066394 (+ 4us) client_negotiation.cc:769] Sending connection context
0621 05:10:23.066448 (+ 54us) client_negotiation.cc:240] Negotiation successful
0621 05:10:23.066511 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35}
I20260621 05:10:23.066623 5228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:10:23.063740 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.131.129:41195 (local address 127.3.131.130:41933)
0621 05:10:23.063973 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0621 05:10:23.063981 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0621 05:10:23.064025 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:10:23.064217 (+ 192us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:10:23.064217 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:10:23.064219 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 05:10:23.064288 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:10:23.064290 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:10:23.065002 (+ 712us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:10:23.065003 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:10:23.065794 (+ 791us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:10:23.065796 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:10:23.066340 (+ 544us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:10:23.066341 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:10:23.066408 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:10:23.066411 (+ 3us) client_negotiation.cc:769] Sending connection context
0621 05:10:23.066486 (+ 75us) client_negotiation.cc:240] Negotiation successful
0621 05:10:23.066525 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":197,"thread_start_us":127,"threads_started":1}
I20260621 05:10:23.067009 5150 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be5721a5e3b5497f95d3d5f7b99ef2c0" candidate_uuid: "98aae82ed2bd426fb5bd4fd0a69ede6f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9643924b7a2495cbae396cd2b2523b4" is_pre_election: true
I20260621 05:10:23.067242 5150 raft_consensus.cc:2454] T be5721a5e3b5497f95d3d5f7b99ef2c0 P b9643924b7a2495cbae396cd2b2523b4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 98aae82ed2bd426fb5bd4fd0a69ede6f in term 0.
I20260621 05:10:23.067260 4870 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be5721a5e3b5497f95d3d5f7b99ef2c0" candidate_uuid: "98aae82ed2bd426fb5bd4fd0a69ede6f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4f7fee4354ab4ed5b1bfae3ec9f66d4a" is_pre_election: true
I20260621 05:10:23.067452 4870 raft_consensus.cc:2454] T be5721a5e3b5497f95d3d5f7b99ef2c0 P 4f7fee4354ab4ed5b1bfae3ec9f66d4a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 98aae82ed2bd426fb5bd4fd0a69ede6f in term 0.
I20260621 05:10:23.067528 4941 leader_election.cc:302] T be5721a5e3b5497f95d3d5f7b99ef2c0 P 98aae82ed2bd426fb5bd4fd0a69ede6f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 98aae82ed2bd426fb5bd4fd0a69ede6f, b9643924b7a2495cbae396cd2b2523b4; no voters:
I20260621 05:10:23.067678 5227 raft_consensus.cc:2781] T be5721a5e3b5497f95d3d5f7b99ef2c0 P 98aae82ed2bd426fb5bd4fd0a69ede6f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260621 05:10:23.067737 5227 raft_consensus.cc:482] T be5721a5e3b5497f95d3d5f7b99ef2c0 P 98aae82ed2bd426fb5bd4fd0a69ede6f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260621 05:10:23.067763 5227 raft_consensus.cc:3037] T be5721a5e3b5497f95d3d5f7b99ef2c0 P 98aae82ed2bd426fb5bd4fd0a69ede6f [term 0 FOLLOWER]: Advancing to term 1
I20260621 05:10:23.068851 5227 raft_consensus.cc:504] T be5721a5e3b5497f95d3d5f7b99ef2c0 P 98aae82ed2bd426fb5bd4fd0a69ede6f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9643924b7a2495cbae396cd2b2523b4" member_type: VOTER last_known_addr { host: "127.3.131.131" port: 45593 } } peers { permanent_uuid: "4f7fee4354ab4ed5b1bfae3ec9f66d4a" member_type: VOTER last_known_addr { host: "127.3.131.129" port: 41195 } } peers { permanent_uuid: "98aae82ed2bd426fb5bd4fd0a69ede6f" member_type: VOTER last_known_addr { host: "127.3.131.130" port: 44133 } }
I20260621 05:10:23.068990 5227 leader_election.cc:288] T be5721a5e3b5497f95d3d5f7b99ef2c0 P 98aae82ed2bd426fb5bd4fd0a69ede6f [CANDIDATE]: Term 1 election: Requested vote from peers b9643924b7a2495cbae396cd2b2523b4 (127.3.131.131:45593), 4f7fee4354ab4ed5b1bfae3ec9f66d4a (127.3.131.129:41195)
I20260621 05:10:23.069197 4870 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be5721a5e3b5497f95d3d5f7b99ef2c0" candidate_uuid: "98aae82ed2bd426fb5bd4fd0a69ede6f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4f7fee4354ab4ed5b1bfae3ec9f66d4a"
I20260621 05:10:23.069278 4870 raft_consensus.cc:3037] T be5721a5e3b5497f95d3d5f7b99ef2c0 P 4f7fee4354ab4ed5b1bfae3ec9f66d4a [term 0 FOLLOWER]: Advancing to term 1
I20260621 05:10:23.069265 5150 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be5721a5e3b5497f95d3d5f7b99ef2c0" candidate_uuid: "98aae82ed2bd426fb5bd4fd0a69ede6f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9643924b7a2495cbae396cd2b2523b4"
I20260621 05:10:23.069397 5150 raft_consensus.cc:3037] T be5721a5e3b5497f95d3d5f7b99ef2c0 P b9643924b7a2495cbae396cd2b2523b4 [term 0 FOLLOWER]: Advancing to term 1
I20260621 05:10:23.070029 4870 raft_consensus.cc:2454] T be5721a5e3b5497f95d3d5f7b99ef2c0 P 4f7fee4354ab4ed5b1bfae3ec9f66d4a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 98aae82ed2bd426fb5bd4fd0a69ede6f in term 1.
I20260621 05:10:23.070197 4941 leader_election.cc:302] T be5721a5e3b5497f95d3d5f7b99ef2c0 P 98aae82ed2bd426fb5bd4fd0a69ede6f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4f7fee4354ab4ed5b1bfae3ec9f66d4a, 98aae82ed2bd426fb5bd4fd0a69ede6f; no voters:
I20260621 05:10:23.070192 5150 raft_consensus.cc:2454] T be5721a5e3b5497f95d3d5f7b99ef2c0 P b9643924b7a2495cbae396cd2b2523b4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 98aae82ed2bd426fb5bd4fd0a69ede6f in term 1.
I20260621 05:10:23.070304 5227 raft_consensus.cc:2781] T be5721a5e3b5497f95d3d5f7b99ef2c0 P 98aae82ed2bd426fb5bd4fd0a69ede6f [term 1 FOLLOWER]: Leader election won for term 1
I20260621 05:10:23.070473 5227 raft_consensus.cc:686] T be5721a5e3b5497f95d3d5f7b99ef2c0 P 98aae82ed2bd426fb5bd4fd0a69ede6f [term 1 LEADER]: Becoming Leader. State: Replica: 98aae82ed2bd426fb5bd4fd0a69ede6f, State: Running, Role: LEADER
I20260621 05:10:23.070556 5227 consensus_queue.cc:237] T be5721a5e3b5497f95d3d5f7b99ef2c0 P 98aae82ed2bd426fb5bd4fd0a69ede6f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9643924b7a2495cbae396cd2b2523b4" member_type: VOTER last_known_addr { host: "127.3.131.131" port: 45593 } } peers { permanent_uuid: "4f7fee4354ab4ed5b1bfae3ec9f66d4a" member_type: VOTER last_known_addr { host: "127.3.131.129" port: 41195 } } peers { permanent_uuid: "98aae82ed2bd426fb5bd4fd0a69ede6f" member_type: VOTER last_known_addr { host: "127.3.131.130" port: 44133 } }
I20260621 05:10:23.071436 4713 catalog_manager.cc:5495] T be5721a5e3b5497f95d3d5f7b99ef2c0 P 98aae82ed2bd426fb5bd4fd0a69ede6f reported cstate change: term changed from 0 to 1, leader changed from <none> to 98aae82ed2bd426fb5bd4fd0a69ede6f (127.3.131.130). New cstate: current_term: 1 leader_uuid: "98aae82ed2bd426fb5bd4fd0a69ede6f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9643924b7a2495cbae396cd2b2523b4" member_type: VOTER last_known_addr { host: "127.3.131.131" port: 45593 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4f7fee4354ab4ed5b1bfae3ec9f66d4a" member_type: VOTER last_known_addr { host: "127.3.131.129" port: 41195 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "98aae82ed2bd426fb5bd4fd0a69ede6f" member_type: VOTER last_known_addr { host: "127.3.131.130" port: 44133 } health_report { overall_health: HEALTHY } } }
I20260621 05:10:23.092466 5217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:10:23.089647 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58812 (local address 127.3.131.130:44133)
0621 05:10:23.089714 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0621 05:10:23.089717 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0621 05:10:23.089727 (+ 10us) server_negotiation.cc:407] Connection header received
0621 05:10:23.089808 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:10:23.089809 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:10:23.089828 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:10:23.089908 (+ 80us) server_negotiation.cc:226] Negotiated authn=TOKEN
0621 05:10:23.090121 (+ 213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:10:23.090632 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:10:23.091528 (+ 896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:10:23.091745 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:10:23.091822 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:10:23.092040 (+ 218us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0621 05:10:23.092178 (+ 138us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0621 05:10:23.092275 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0621 05:10:23.092321 (+ 46us) server_negotiation.cc:299] Negotiation successful
0621 05:10:23.092354 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260621 05:10:23.094465 4870 raft_consensus.cc:1261] T be5721a5e3b5497f95d3d5f7b99ef2c0 P 4f7fee4354ab4ed5b1bfae3ec9f66d4a [term 1 FOLLOWER]: Refusing update from remote peer 98aae82ed2bd426fb5bd4fd0a69ede6f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 05:10:23.094465 5150 raft_consensus.cc:1261] T be5721a5e3b5497f95d3d5f7b99ef2c0 P b9643924b7a2495cbae396cd2b2523b4 [term 1 FOLLOWER]: Refusing update from remote peer 98aae82ed2bd426fb5bd4fd0a69ede6f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 05:10:23.094852 5227 consensus_queue.cc:1035] T be5721a5e3b5497f95d3d5f7b99ef2c0 P 98aae82ed2bd426fb5bd4fd0a69ede6f [LEADER]: Connected to new peer: Peer: permanent_uuid: "4f7fee4354ab4ed5b1bfae3ec9f66d4a" member_type: VOTER last_known_addr { host: "127.3.131.129" port: 41195 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 05:10:23.095108 5229 consensus_queue.cc:1035] T be5721a5e3b5497f95d3d5f7b99ef2c0 P 98aae82ed2bd426fb5bd4fd0a69ede6f [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9643924b7a2495cbae396cd2b2523b4" member_type: VOTER last_known_addr { host: "127.3.131.131" port: 45593 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 05:10:23.097883 5233 mvcc.cc:205] Tried to move back new op lower bound from 7299148280191455232 to 7299148280097726464. Current Snapshot: MvccSnapshot[applied={T|T < 7299148280191455232}]
I20260621 05:10:23.098174 5234 mvcc.cc:205] Tried to move back new op lower bound from 7299148280191455232 to 7299148280097726464. Current Snapshot: MvccSnapshot[applied={T|T < 7299148280191455232}]
I20260621 05:10:23.101708 4712 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46762:
table { table_name: "test-table" } modify_external_catalogs: true
I20260621 05:10:23.101833 4712 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46762:
table { table_name: "test-table" } modify_external_catalogs: true
I20260621 05:10:23.102689 4712 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d11704c4279f426fa489c4af98674cd5: Sending DeleteTablet for 3 replicas of tablet be5721a5e3b5497f95d3d5f7b99ef2c0
I20260621 05:10:23.103152 4850 tablet_service.cc:1514] Processing DeleteTablet for tablet be5721a5e3b5497f95d3d5f7b99ef2c0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 05:10:23 UTC) from {username='kudu', principal='kudu/127.3.131.190@KRBTEST.COM'} at 127.0.0.1:51878
I20260621 05:10:23.103164 4990 tablet_service.cc:1514] Processing DeleteTablet for tablet be5721a5e3b5497f95d3d5f7b99ef2c0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 05:10:23 UTC) from {username='kudu', principal='kudu/127.3.131.190@KRBTEST.COM'} at 127.0.0.1:58808
I20260621 05:10:23.103442 5130 tablet_service.cc:1514] Processing DeleteTablet for tablet be5721a5e3b5497f95d3d5f7b99ef2c0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 05:10:23 UTC) from {username='kudu', principal='kudu/127.3.131.190@KRBTEST.COM'} at 127.0.0.1:54090
I20260621 05:10:23.103487 5243 tablet_replica.cc:330] T be5721a5e3b5497f95d3d5f7b99ef2c0 P 98aae82ed2bd426fb5bd4fd0a69ede6f: stopping tablet replica
I20260621 05:10:23.103588 5243 raft_consensus.cc:2229] T be5721a5e3b5497f95d3d5f7b99ef2c0 P 98aae82ed2bd426fb5bd4fd0a69ede6f [term 1 LEADER]: Raft consensus shutting down.
I20260621 05:10:23.103749 5243 raft_consensus.cc:2258] T be5721a5e3b5497f95d3d5f7b99ef2c0 P 98aae82ed2bd426fb5bd4fd0a69ede6f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 05:10:23.103790 4712 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46762:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260621 05:10:23.103950 4712 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260621 05:10:23.104243 5243 ts_tablet_manager.cc:1905] T be5721a5e3b5497f95d3d5f7b99ef2c0 P 98aae82ed2bd426fb5bd4fd0a69ede6f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 05:10:23.104250 5244 tablet_replica.cc:330] T be5721a5e3b5497f95d3d5f7b99ef2c0 P b9643924b7a2495cbae396cd2b2523b4: stopping tablet replica
I20260621 05:10:23.105329 4990 tablet_service.cc:1467] Processing CreateTablet for tablet 1440e0f2d5aa4770a628324cebdb279d (DEFAULT_TABLE table=test-table [id=141d3bfb2af14566b40b4f7a3f685a03]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 05:10:23.105463 4990 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1440e0f2d5aa4770a628324cebdb279d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 05:10:23.106134 5243 ts_tablet_manager.cc:1918] T be5721a5e3b5497f95d3d5f7b99ef2c0 P 98aae82ed2bd426fb5bd4fd0a69ede6f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260621 05:10:23.106241 5243 log.cc:1196] T be5721a5e3b5497f95d3d5f7b99ef2c0 P 98aae82ed2bd426fb5bd4fd0a69ede6f: Deleting WAL directory at /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782018616648578-3598-0/minicluster-data/ts-1/wal/wals/be5721a5e3b5497f95d3d5f7b99ef2c0
I20260621 05:10:23.106496 5243 ts_tablet_manager.cc:1939] T be5721a5e3b5497f95d3d5f7b99ef2c0 P 98aae82ed2bd426fb5bd4fd0a69ede6f: Deleting consensus metadata
I20260621 05:10:23.106882 5220 tablet_bootstrap.cc:492] T 1440e0f2d5aa4770a628324cebdb279d P 98aae82ed2bd426fb5bd4fd0a69ede6f: Bootstrap starting.
I20260621 05:10:23.106940 4850 tablet_service.cc:1467] Processing CreateTablet for tablet 1440e0f2d5aa4770a628324cebdb279d (DEFAULT_TABLE table=test-table [id=141d3bfb2af14566b40b4f7a3f685a03]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 05:10:23.107071 4850 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1440e0f2d5aa4770a628324cebdb279d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 05:10:23.107331 5130 tablet_service.cc:1467] Processing CreateTablet for tablet 1440e0f2d5aa4770a628324cebdb279d (DEFAULT_TABLE table=test-table [id=141d3bfb2af14566b40b4f7a3f685a03]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 05:10:23.107410 5220 tablet_bootstrap.cc:654] T 1440e0f2d5aa4770a628324cebdb279d P 98aae82ed2bd426fb5bd4fd0a69ede6f: Neither blocks nor log segments found. Creating new log.
I20260621 05:10:23.107460 5130 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1440e0f2d5aa4770a628324cebdb279d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 05:10:23.108757 5221 tablet_bootstrap.cc:492] T 1440e0f2d5aa4770a628324cebdb279d P 4f7fee4354ab4ed5b1bfae3ec9f66d4a: Bootstrap starting.
I20260621 05:10:23.109028 5219 tablet_bootstrap.cc:492] T 1440e0f2d5aa4770a628324cebdb279d P b9643924b7a2495cbae396cd2b2523b4: Bootstrap starting.
I20260621 05:10:23.109205 5221 tablet_bootstrap.cc:654] T 1440e0f2d5aa4770a628324cebdb279d P 4f7fee4354ab4ed5b1bfae3ec9f66d4a: Neither blocks nor log segments found. Creating new log.
I20260621 05:10:23.109237 5220 tablet_bootstrap.cc:492] T 1440e0f2d5aa4770a628324cebdb279d P 98aae82ed2bd426fb5bd4fd0a69ede6f: No bootstrap required, opened a new log
I20260621 05:10:23.109282 5220 ts_tablet_manager.cc:1397] T 1440e0f2d5aa4770a628324cebdb279d P 98aae82ed2bd426fb5bd4fd0a69ede6f: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260621 05:10:23.109431 5220 raft_consensus.cc:348] T 1440e0f2d5aa4770a628324cebdb279d P 98aae82ed2bd426fb5bd4fd0a69ede6f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f7fee4354ab4ed5b1bfae3ec9f66d4a" member_type: VOTER last_known_addr { host: "127.3.131.129" port: 41195 } } peers { permanent_uuid: "98aae82ed2bd426fb5bd4fd0a69ede6f" member_type: VOTER last_known_addr { host: "127.3.131.130" port: 44133 } } peers { permanent_uuid: "b9643924b7a2495cbae396cd2b2523b4" member_type: VOTER last_known_addr { host: "127.3.131.131" port: 45593 } }
I20260621 05:10:23.109474 5219 tablet_bootstrap.cc:654] T 1440e0f2d5aa4770a628324cebdb279d P b9643924b7a2495cbae396cd2b2523b4: Neither blocks nor log segments found. Creating new log.
I20260621 05:10:23.109500 5220 raft_consensus.cc:374] T 1440e0f2d5aa4770a628324cebdb279d P 98aae82ed2bd426fb5bd4fd0a69ede6f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 05:10:23.109521 5220 raft_consensus.cc:729] T 1440e0f2d5aa4770a628324cebdb279d P 98aae82ed2bd426fb5bd4fd0a69ede6f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98aae82ed2bd426fb5bd4fd0a69ede6f, State: Initialized, Role: FOLLOWER
I20260621 05:10:23.109597 5220 consensus_queue.cc:260] T 1440e0f2d5aa4770a628324cebdb279d P 98aae82ed2bd426fb5bd4fd0a69ede6f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f7fee4354ab4ed5b1bfae3ec9f66d4a" member_type: VOTER last_known_addr { host: "127.3.131.129" port: 41195 } } peers { permanent_uuid: "98aae82ed2bd426fb5bd4fd0a69ede6f" member_type: VOTER last_known_addr { host: "127.3.131.130" port: 44133 } } peers { permanent_uuid: "b9643924b7a2495cbae396cd2b2523b4" member_type: VOTER last_known_addr { host: "127.3.131.131" port: 45593 } }
I20260621 05:10:23.109699 5220 ts_tablet_manager.cc:1428] T 1440e0f2d5aa4770a628324cebdb279d P 98aae82ed2bd426fb5bd4fd0a69ede6f: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260621 05:10:23.109936 4699 catalog_manager.cc:4841] TS 98aae82ed2bd426fb5bd4fd0a69ede6f (127.3.131.130:44133): tablet be5721a5e3b5497f95d3d5f7b99ef2c0 (table test-table [id=40f6a3e9f12d4bd8bac69db474d0f59e]) successfully deleted
I20260621 05:10:23.110234 5244 raft_consensus.cc:2229] T be5721a5e3b5497f95d3d5f7b99ef2c0 P b9643924b7a2495cbae396cd2b2523b4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260621 05:10:23.110311 5221 tablet_bootstrap.cc:492] T 1440e0f2d5aa4770a628324cebdb279d P 4f7fee4354ab4ed5b1bfae3ec9f66d4a: No bootstrap required, opened a new log
I20260621 05:10:23.110373 5221 ts_tablet_manager.cc:1397] T 1440e0f2d5aa4770a628324cebdb279d P 4f7fee4354ab4ed5b1bfae3ec9f66d4a: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260621 05:10:23.110461 5244 raft_consensus.cc:2258] T be5721a5e3b5497f95d3d5f7b99ef2c0 P b9643924b7a2495cbae396cd2b2523b4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 05:10:23.110534 5221 raft_consensus.cc:348] T 1440e0f2d5aa4770a628324cebdb279d P 4f7fee4354ab4ed5b1bfae3ec9f66d4a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f7fee4354ab4ed5b1bfae3ec9f66d4a" member_type: VOTER last_known_addr { host: "127.3.131.129" port: 41195 } } peers { permanent_uuid: "98aae82ed2bd426fb5bd4fd0a69ede6f" member_type: VOTER last_known_addr { host: "127.3.131.130" port: 44133 } } peers { permanent_uuid: "b9643924b7a2495cbae396cd2b2523b4" member_type: VOTER last_known_addr { host: "127.3.131.131" port: 45593 } }
I20260621 05:10:23.110625 5221 raft_consensus.cc:374] T 1440e0f2d5aa4770a628324cebdb279d P 4f7fee4354ab4ed5b1bfae3ec9f66d4a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 05:10:23.110656 5221 raft_consensus.cc:729] T 1440e0f2d5aa4770a628324cebdb279d P 4f7fee4354ab4ed5b1bfae3ec9f66d4a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4f7fee4354ab4ed5b1bfae3ec9f66d4a, State: Initialized, Role: FOLLOWER
I20260621 05:10:23.110719 5221 consensus_queue.cc:260] T 1440e0f2d5aa4770a628324cebdb279d P 4f7fee4354ab4ed5b1bfae3ec9f66d4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f7fee4354ab4ed5b1bfae3ec9f66d4a" member_type: VOTER last_known_addr { host: "127.3.131.129" port: 41195 } } peers { permanent_uuid: "98aae82ed2bd426fb5bd4fd0a69ede6f" member_type: VOTER last_known_addr { host: "127.3.131.130" port: 44133 } } peers { permanent_uuid: "b9643924b7a2495cbae396cd2b2523b4" member_type: VOTER last_known_addr { host: "127.3.131.131" port: 45593 } }
I20260621 05:10:23.110836 5221 ts_tablet_manager.cc:1428] T 1440e0f2d5aa4770a628324cebdb279d P 4f7fee4354ab4ed5b1bfae3ec9f66d4a: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260621 05:10:23.110723 5242 tablet_replica.cc:330] T be5721a5e3b5497f95d3d5f7b99ef2c0 P 4f7fee4354ab4ed5b1bfae3ec9f66d4a: stopping tablet replica
I20260621 05:10:23.111291 5242 raft_consensus.cc:2229] T be5721a5e3b5497f95d3d5f7b99ef2c0 P 4f7fee4354ab4ed5b1bfae3ec9f66d4a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260621 05:10:23.111577 5244 ts_tablet_manager.cc:1905] T be5721a5e3b5497f95d3d5f7b99ef2c0 P b9643924b7a2495cbae396cd2b2523b4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 05:10:23.111655 5242 raft_consensus.cc:2258] T be5721a5e3b5497f95d3d5f7b99ef2c0 P 4f7fee4354ab4ed5b1bfae3ec9f66d4a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 05:10:23.112171 5219 tablet_bootstrap.cc:492] T 1440e0f2d5aa4770a628324cebdb279d P b9643924b7a2495cbae396cd2b2523b4: No bootstrap required, opened a new log
I20260621 05:10:23.112227 5219 ts_tablet_manager.cc:1397] T 1440e0f2d5aa4770a628324cebdb279d P b9643924b7a2495cbae396cd2b2523b4: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260621 05:10:23.112368 5219 raft_consensus.cc:348] T 1440e0f2d5aa4770a628324cebdb279d P b9643924b7a2495cbae396cd2b2523b4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f7fee4354ab4ed5b1bfae3ec9f66d4a" member_type: VOTER last_known_addr { host: "127.3.131.129" port: 41195 } } peers { permanent_uuid: "98aae82ed2bd426fb5bd4fd0a69ede6f" member_type: VOTER last_known_addr { host: "127.3.131.130" port: 44133 } } peers { permanent_uuid: "b9643924b7a2495cbae396cd2b2523b4" member_type: VOTER last_known_addr { host: "127.3.131.131" port: 45593 } }
I20260621 05:10:23.112443 5219 raft_consensus.cc:374] T 1440e0f2d5aa4770a628324cebdb279d P b9643924b7a2495cbae396cd2b2523b4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 05:10:23.112466 5219 raft_consensus.cc:729] T 1440e0f2d5aa4770a628324cebdb279d P b9643924b7a2495cbae396cd2b2523b4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b9643924b7a2495cbae396cd2b2523b4, State: Initialized, Role: FOLLOWER
I20260621 05:10:23.112510 5219 consensus_queue.cc:260] T 1440e0f2d5aa4770a628324cebdb279d P b9643924b7a2495cbae396cd2b2523b4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f7fee4354ab4ed5b1bfae3ec9f66d4a" member_type: VOTER last_known_addr { host: "127.3.131.129" port: 41195 } } peers { permanent_uuid: "98aae82ed2bd426fb5bd4fd0a69ede6f" member_type: VOTER last_known_addr { host: "127.3.131.130" port: 44133 } } peers { permanent_uuid: "b9643924b7a2495cbae396cd2b2523b4" member_type: VOTER last_known_addr { host: "127.3.131.131" port: 45593 } }
I20260621 05:10:23.112604 5219 ts_tablet_manager.cc:1428] T 1440e0f2d5aa4770a628324cebdb279d P b9643924b7a2495cbae396cd2b2523b4: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260621 05:10:23.113088 5242 ts_tablet_manager.cc:1905] T be5721a5e3b5497f95d3d5f7b99ef2c0 P 4f7fee4354ab4ed5b1bfae3ec9f66d4a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 05:10:23.113382 5244 ts_tablet_manager.cc:1918] T be5721a5e3b5497f95d3d5f7b99ef2c0 P b9643924b7a2495cbae396cd2b2523b4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260621 05:10:23.113461 5244 log.cc:1196] T be5721a5e3b5497f95d3d5f7b99ef2c0 P b9643924b7a2495cbae396cd2b2523b4: Deleting WAL directory at /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782018616648578-3598-0/minicluster-data/ts-2/wal/wals/be5721a5e3b5497f95d3d5f7b99ef2c0
I20260621 05:10:23.113806 5244 ts_tablet_manager.cc:1939] T be5721a5e3b5497f95d3d5f7b99ef2c0 P b9643924b7a2495cbae396cd2b2523b4: Deleting consensus metadata
I20260621 05:10:23.114267 4697 catalog_manager.cc:4841] TS b9643924b7a2495cbae396cd2b2523b4 (127.3.131.131:45593): tablet be5721a5e3b5497f95d3d5f7b99ef2c0 (table test-table [id=40f6a3e9f12d4bd8bac69db474d0f59e]) successfully deleted
I20260621 05:10:23.114909 5242 ts_tablet_manager.cc:1918] T be5721a5e3b5497f95d3d5f7b99ef2c0 P 4f7fee4354ab4ed5b1bfae3ec9f66d4a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260621 05:10:23.115005 5242 log.cc:1196] T be5721a5e3b5497f95d3d5f7b99ef2c0 P 4f7fee4354ab4ed5b1bfae3ec9f66d4a: Deleting WAL directory at /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782018616648578-3598-0/minicluster-data/ts-0/wal/wals/be5721a5e3b5497f95d3d5f7b99ef2c0
I20260621 05:10:23.115309 5242 ts_tablet_manager.cc:1939] T be5721a5e3b5497f95d3d5f7b99ef2c0 P 4f7fee4354ab4ed5b1bfae3ec9f66d4a: Deleting consensus metadata
I20260621 05:10:23.115896 4697 catalog_manager.cc:4841] TS 4f7fee4354ab4ed5b1bfae3ec9f66d4a (127.3.131.129:41195): tablet be5721a5e3b5497f95d3d5f7b99ef2c0 (table test-table [id=40f6a3e9f12d4bd8bac69db474d0f59e]) successfully deleted
W20260621 05:10:23.139209 4917 tablet.cc:2369] T 1440e0f2d5aa4770a628324cebdb279d P 4f7fee4354ab4ed5b1bfae3ec9f66d4a: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260621 05:10:23.221966 5197 tablet.cc:2369] T 1440e0f2d5aa4770a628324cebdb279d P b9643924b7a2495cbae396cd2b2523b4: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260621 05:10:23.299152 5226 raft_consensus.cc:482] T 1440e0f2d5aa4770a628324cebdb279d P b9643924b7a2495cbae396cd2b2523b4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260621 05:10:23.299292 5226 raft_consensus.cc:504] T 1440e0f2d5aa4770a628324cebdb279d P b9643924b7a2495cbae396cd2b2523b4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f7fee4354ab4ed5b1bfae3ec9f66d4a" member_type: VOTER last_known_addr { host: "127.3.131.129" port: 41195 } } peers { permanent_uuid: "98aae82ed2bd426fb5bd4fd0a69ede6f" member_type: VOTER last_known_addr { host: "127.3.131.130" port: 44133 } } peers { permanent_uuid: "b9643924b7a2495cbae396cd2b2523b4" member_type: VOTER last_known_addr { host: "127.3.131.131" port: 45593 } }
I20260621 05:10:23.299758 5226 leader_election.cc:288] T 1440e0f2d5aa4770a628324cebdb279d P b9643924b7a2495cbae396cd2b2523b4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4f7fee4354ab4ed5b1bfae3ec9f66d4a (127.3.131.129:41195), 98aae82ed2bd426fb5bd4fd0a69ede6f (127.3.131.130:44133)
I20260621 05:10:23.303990 5089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:10:23.299952 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.131.129:41195 (local address 127.3.131.131:58723)
0621 05:10:23.299993 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0621 05:10:23.300005 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0621 05:10:23.300059 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:10:23.300359 (+ 300us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:10:23.300360 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:10:23.300362 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 05:10:23.300435 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:10:23.300437 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:10:23.301363 (+ 926us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:10:23.301363 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:10:23.302851 (+ 1488us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:10:23.302855 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:10:23.303682 (+ 827us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:10:23.303683 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:10:23.303779 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:10:23.303784 (+ 5us) client_negotiation.cc:769] Sending connection context
0621 05:10:23.303856 (+ 72us) client_negotiation.cc:240] Negotiation successful
0621 05:10:23.303902 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":19}
I20260621 05:10:23.304005 5218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:10:23.300152 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.131.131:58723 (local address 127.3.131.129:41195)
0621 05:10:23.300195 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0621 05:10:23.300197 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:10:23.300206 (+ 9us) server_negotiation.cc:407] Connection header received
0621 05:10:23.300236 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:10:23.300236 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:10:23.300247 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:10:23.300299 (+ 52us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 05:10:23.300514 (+ 215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:10:23.301270 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:10:23.302946 (+ 1676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:10:23.303572 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:10:23.303635 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:10:23.303643 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0621 05:10:23.303912 (+ 269us) server_negotiation.cc:299] Negotiation successful
0621 05:10:23.303949 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260621 05:10:23.304463 4870 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1440e0f2d5aa4770a628324cebdb279d" candidate_uuid: "b9643924b7a2495cbae396cd2b2523b4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4f7fee4354ab4ed5b1bfae3ec9f66d4a" is_pre_election: true
I20260621 05:10:23.304564 4870 raft_consensus.cc:2454] T 1440e0f2d5aa4770a628324cebdb279d P 4f7fee4354ab4ed5b1bfae3ec9f66d4a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9643924b7a2495cbae396cd2b2523b4 in term 0.
I20260621 05:10:23.304538 5248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:10:23.299976 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.131.130:44133 (local address 127.3.131.131:60029)
0621 05:10:23.300260 (+ 284us) negotiation.cc:107] Waiting for socket to connect
0621 05:10:23.300270 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0621 05:10:23.300346 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:10:23.300611 (+ 265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:10:23.300612 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:10:23.300615 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 05:10:23.300704 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:10:23.300706 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:10:23.301612 (+ 906us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:10:23.301613 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:10:23.303318 (+ 1705us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:10:23.303321 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:10:23.304273 (+ 952us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:10:23.304274 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:10:23.304354 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:10:23.304358 (+ 4us) client_negotiation.cc:769] Sending connection context
0621 05:10:23.304422 (+ 64us) client_negotiation.cc:240] Negotiation successful
0621 05:10:23.304470 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":271,"thread_start_us":86,"threads_started":1}
I20260621 05:10:23.304778 5217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:10:23.300152 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.131.131:60029 (local address 127.3.131.130:44133)
0621 05:10:23.300283 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0621 05:10:23.300285 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:10:23.300320 (+ 35us) server_negotiation.cc:407] Connection header received
0621 05:10:23.300452 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:10:23.300453 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:10:23.300469 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:10:23.300542 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 05:10:23.300786 (+ 244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:10:23.301497 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:10:23.303436 (+ 1939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:10:23.303988 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:10:23.304062 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:10:23.304071 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0621 05:10:23.304661 (+ 590us) server_negotiation.cc:299] Negotiation successful
0621 05:10:23.304712 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260621 05:10:23.304832 5081 leader_election.cc:302] T 1440e0f2d5aa4770a628324cebdb279d P b9643924b7a2495cbae396cd2b2523b4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4f7fee4354ab4ed5b1bfae3ec9f66d4a, b9643924b7a2495cbae396cd2b2523b4; no voters:
I20260621 05:10:23.304946 5226 raft_consensus.cc:2781] T 1440e0f2d5aa4770a628324cebdb279d P b9643924b7a2495cbae396cd2b2523b4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260621 05:10:23.304991 5226 raft_consensus.cc:482] T 1440e0f2d5aa4770a628324cebdb279d P b9643924b7a2495cbae396cd2b2523b4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260621 05:10:23.305017 5226 raft_consensus.cc:3037] T 1440e0f2d5aa4770a628324cebdb279d P b9643924b7a2495cbae396cd2b2523b4 [term 0 FOLLOWER]: Advancing to term 1
I20260621 05:10:23.305099 5010 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1440e0f2d5aa4770a628324cebdb279d" candidate_uuid: "b9643924b7a2495cbae396cd2b2523b4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98aae82ed2bd426fb5bd4fd0a69ede6f" is_pre_election: true
I20260621 05:10:23.305218 5010 raft_consensus.cc:2454] T 1440e0f2d5aa4770a628324cebdb279d P 98aae82ed2bd426fb5bd4fd0a69ede6f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9643924b7a2495cbae396cd2b2523b4 in term 0.
I20260621 05:10:23.305675 5226 raft_consensus.cc:504] T 1440e0f2d5aa4770a628324cebdb279d P b9643924b7a2495cbae396cd2b2523b4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f7fee4354ab4ed5b1bfae3ec9f66d4a" member_type: VOTER last_known_addr { host: "127.3.131.129" port: 41195 } } peers { permanent_uuid: "98aae82ed2bd426fb5bd4fd0a69ede6f" member_type: VOTER last_known_addr { host: "127.3.131.130" port: 44133 } } peers { permanent_uuid: "b9643924b7a2495cbae396cd2b2523b4" member_type: VOTER last_known_addr { host: "127.3.131.131" port: 45593 } }
I20260621 05:10:23.305828 5226 leader_election.cc:288] T 1440e0f2d5aa4770a628324cebdb279d P b9643924b7a2495cbae396cd2b2523b4 [CANDIDATE]: Term 1 election: Requested vote from peers 4f7fee4354ab4ed5b1bfae3ec9f66d4a (127.3.131.129:41195), 98aae82ed2bd426fb5bd4fd0a69ede6f (127.3.131.130:44133)
I20260621 05:10:23.305979 4870 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1440e0f2d5aa4770a628324cebdb279d" candidate_uuid: "b9643924b7a2495cbae396cd2b2523b4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4f7fee4354ab4ed5b1bfae3ec9f66d4a"
I20260621 05:10:23.306064 4870 raft_consensus.cc:3037] T 1440e0f2d5aa4770a628324cebdb279d P 4f7fee4354ab4ed5b1bfae3ec9f66d4a [term 0 FOLLOWER]: Advancing to term 1
I20260621 05:10:23.306684 4870 raft_consensus.cc:2454] T 1440e0f2d5aa4770a628324cebdb279d P 4f7fee4354ab4ed5b1bfae3ec9f66d4a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9643924b7a2495cbae396cd2b2523b4 in term 1.
I20260621 05:10:23.306869 5081 leader_election.cc:302] T 1440e0f2d5aa4770a628324cebdb279d P b9643924b7a2495cbae396cd2b2523b4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4f7fee4354ab4ed5b1bfae3ec9f66d4a, b9643924b7a2495cbae396cd2b2523b4; no voters:
I20260621 05:10:23.306866 5010 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1440e0f2d5aa4770a628324cebdb279d" candidate_uuid: "b9643924b7a2495cbae396cd2b2523b4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98aae82ed2bd426fb5bd4fd0a69ede6f"
I20260621 05:10:23.306963 5226 raft_consensus.cc:2781] T 1440e0f2d5aa4770a628324cebdb279d P b9643924b7a2495cbae396cd2b2523b4 [term 1 FOLLOWER]: Leader election won for term 1
I20260621 05:10:23.306982 5010 raft_consensus.cc:3037] T 1440e0f2d5aa4770a628324cebdb279d P 98aae82ed2bd426fb5bd4fd0a69ede6f [term 0 FOLLOWER]: Advancing to term 1
I20260621 05:10:23.307081 5226 raft_consensus.cc:686] T 1440e0f2d5aa4770a628324cebdb279d P b9643924b7a2495cbae396cd2b2523b4 [term 1 LEADER]: Becoming Leader. State: Replica: b9643924b7a2495cbae396cd2b2523b4, State: Running, Role: LEADER
I20260621 05:10:23.307153 5226 consensus_queue.cc:237] T 1440e0f2d5aa4770a628324cebdb279d P b9643924b7a2495cbae396cd2b2523b4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f7fee4354ab4ed5b1bfae3ec9f66d4a" member_type: VOTER last_known_addr { host: "127.3.131.129" port: 41195 } } peers { permanent_uuid: "98aae82ed2bd426fb5bd4fd0a69ede6f" member_type: VOTER last_known_addr { host: "127.3.131.130" port: 44133 } } peers { permanent_uuid: "b9643924b7a2495cbae396cd2b2523b4" member_type: VOTER last_known_addr { host: "127.3.131.131" port: 45593 } }
I20260621 05:10:23.307864 5010 raft_consensus.cc:2454] T 1440e0f2d5aa4770a628324cebdb279d P 98aae82ed2bd426fb5bd4fd0a69ede6f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9643924b7a2495cbae396cd2b2523b4 in term 1.
I20260621 05:10:23.308445 4712 catalog_manager.cc:5495] T 1440e0f2d5aa4770a628324cebdb279d P b9643924b7a2495cbae396cd2b2523b4 reported cstate change: term changed from 0 to 1, leader changed from <none> to b9643924b7a2495cbae396cd2b2523b4 (127.3.131.131). New cstate: current_term: 1 leader_uuid: "b9643924b7a2495cbae396cd2b2523b4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f7fee4354ab4ed5b1bfae3ec9f66d4a" member_type: VOTER last_known_addr { host: "127.3.131.129" port: 41195 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "98aae82ed2bd426fb5bd4fd0a69ede6f" member_type: VOTER last_known_addr { host: "127.3.131.130" port: 44133 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b9643924b7a2495cbae396cd2b2523b4" member_type: VOTER last_known_addr { host: "127.3.131.131" port: 45593 } health_report { overall_health: HEALTHY } } }
Jun 21 05:10:23 dist-test-slave-vrq1 krb5kdc[4668](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782018622, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.131.190@KRBTEST.COM for kudu/127.3.131.190@KRBTEST.COM
I20260621 05:10:23.371367 5256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:10:23.363978 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.131.190:43199 (local address 127.0.0.1:46778)
0621 05:10:23.364185 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0621 05:10:23.364196 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0621 05:10:23.364264 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:10:23.364396 (+ 132us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:10:23.364397 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:10:23.364706 (+ 309us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:10:23.364762 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:10:23.364764 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:10:23.365408 (+ 644us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:10:23.365409 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:10:23.365957 (+ 548us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:10:23.365959 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:10:23.366422 (+ 463us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:10:23.366424 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:10:23.366472 (+ 48us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:10:23.367069 (+ 597us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 05:10:23.367074 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:10:23.368527 (+ 1453us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 05:10:23.370143 (+ 1616us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:10:23.370150 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:10:23.370152 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:10:23.370358 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:10:23.370576 (+ 218us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:10:23.370577 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:10:23.370578 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:10:23.370646 (+ 68us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:10:23.371004 (+ 358us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 05:10:23.371010 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 05:10:23.371055 (+ 45us) client_negotiation.cc:769] Sending connection context
0621 05:10:23.371136 (+ 81us) client_negotiation.cc:240] Negotiation successful
0621 05:10:23.371291 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":145,"threads_started":1}
I20260621 05:10:23.371609 4763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:10:23.364060 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46778 (local address 127.3.131.190:43199)
0621 05:10:23.364171 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0621 05:10:23.364174 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0621 05:10:23.364263 (+ 89us) server_negotiation.cc:407] Connection header received
0621 05:10:23.364322 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:10:23.364323 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:10:23.364333 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:10:23.364375 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:10:23.364829 (+ 454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:10:23.365307 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:10:23.366049 (+ 742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:10:23.366336 (+ 287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:10:23.366393 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:10:23.368648 (+ 2255us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:10:23.368649 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:10:23.368649 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:10:23.368655 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:10:23.370043 (+ 1388us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:10:23.370433 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:10:23.370434 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:10:23.370434 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:10:23.370488 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:10:23.370784 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:10:23.370785 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:10:23.370785 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:10:23.370914 (+ 129us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:10:23.370971 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0621 05:10:23.371438 (+ 467us) server_negotiation.cc:299] Negotiation successful
0621 05:10:23.371536 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260621 05:10:23.373606 4712 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.3.131.190@KRBTEST.COM'} at 127.0.0.1:46778:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260621 05:10:23.373822 4712 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260621 05:10:23.375339 4850 tablet_service.cc:1467] Processing CreateTablet for tablet 2a9c9cc782ee48bc8c64b9fb93db7ea0 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=90a44a2949524466b911b94da0a16a09]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260621 05:10:23.375372 5130 tablet_service.cc:1467] Processing CreateTablet for tablet 2a9c9cc782ee48bc8c64b9fb93db7ea0 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=90a44a2949524466b911b94da0a16a09]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260621 05:10:23.375389 4990 tablet_service.cc:1467] Processing CreateTablet for tablet 2a9c9cc782ee48bc8c64b9fb93db7ea0 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=90a44a2949524466b911b94da0a16a09]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260621 05:10:23.375514 5130 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2a9c9cc782ee48bc8c64b9fb93db7ea0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 05:10:23.375514 4850 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2a9c9cc782ee48bc8c64b9fb93db7ea0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 05:10:23.375551 4990 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2a9c9cc782ee48bc8c64b9fb93db7ea0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 05:10:23.376731 5220 tablet_bootstrap.cc:492] T 2a9c9cc782ee48bc8c64b9fb93db7ea0 P 98aae82ed2bd426fb5bd4fd0a69ede6f: Bootstrap starting.
I20260621 05:10:23.377091 5221 tablet_bootstrap.cc:492] T 2a9c9cc782ee48bc8c64b9fb93db7ea0 P 4f7fee4354ab4ed5b1bfae3ec9f66d4a: Bootstrap starting.
I20260621 05:10:23.377161 5220 tablet_bootstrap.cc:654] T 2a9c9cc782ee48bc8c64b9fb93db7ea0 P 98aae82ed2bd426fb5bd4fd0a69ede6f: Neither blocks nor log segments found. Creating new log.
I20260621 05:10:23.377528 5221 tablet_bootstrap.cc:654] T 2a9c9cc782ee48bc8c64b9fb93db7ea0 P 4f7fee4354ab4ed5b1bfae3ec9f66d4a: Neither blocks nor log segments found. Creating new log.
I20260621 05:10:23.377823 5220 tablet_bootstrap.cc:492] T 2a9c9cc782ee48bc8c64b9fb93db7ea0 P 98aae82ed2bd426fb5bd4fd0a69ede6f: No bootstrap required, opened a new log
I20260621 05:10:23.377890 5220 ts_tablet_manager.cc:1397] T 2a9c9cc782ee48bc8c64b9fb93db7ea0 P 98aae82ed2bd426fb5bd4fd0a69ede6f: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260621 05:10:23.377916 5219 tablet_bootstrap.cc:492] T 2a9c9cc782ee48bc8c64b9fb93db7ea0 P b9643924b7a2495cbae396cd2b2523b4: Bootstrap starting.
I20260621 05:10:23.378008 5220 raft_consensus.cc:348] T 2a9c9cc782ee48bc8c64b9fb93db7ea0 P 98aae82ed2bd426fb5bd4fd0a69ede6f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f7fee4354ab4ed5b1bfae3ec9f66d4a" member_type: VOTER last_known_addr { host: "127.3.131.129" port: 41195 } } peers { permanent_uuid: "98aae82ed2bd426fb5bd4fd0a69ede6f" member_type: VOTER last_known_addr { host: "127.3.131.130" port: 44133 } } peers { permanent_uuid: "b9643924b7a2495cbae396cd2b2523b4" member_type: VOTER last_known_addr { host: "127.3.131.131" port: 45593 } }
I20260621 05:10:23.378114 5220 raft_consensus.cc:374] T 2a9c9cc782ee48bc8c64b9fb93db7ea0 P 98aae82ed2bd426fb5bd4fd0a69ede6f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 05:10:23.378141 5220 raft_consensus.cc:729] T 2a9c9cc782ee48bc8c64b9fb93db7ea0 P 98aae82ed2bd426fb5bd4fd0a69ede6f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98aae82ed2bd426fb5bd4fd0a69ede6f, State: Initialized, Role: FOLLOWER
I20260621 05:10:23.378208 5221 tablet_bootstrap.cc:492] T 2a9c9cc782ee48bc8c64b9fb93db7ea0 P 4f7fee4354ab4ed5b1bfae3ec9f66d4a: No bootstrap required, opened a new log
I20260621 05:10:23.378249 5221 ts_tablet_manager.cc:1397] T 2a9c9cc782ee48bc8c64b9fb93db7ea0 P 4f7fee4354ab4ed5b1bfae3ec9f66d4a: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260621 05:10:23.378193 5220 consensus_queue.cc:260] T 2a9c9cc782ee48bc8c64b9fb93db7ea0 P 98aae82ed2bd426fb5bd4fd0a69ede6f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f7fee4354ab4ed5b1bfae3ec9f66d4a" member_type: VOTER last_known_addr { host: "127.3.131.129" port: 41195 } } peers { permanent_uuid: "98aae82ed2bd426fb5bd4fd0a69ede6f" member_type: VOTER last_known_addr { host: "127.3.131.130" port: 44133 } } peers { permanent_uuid: "b9643924b7a2495cbae396cd2b2523b4" member_type: VOTER last_known_addr { host: "127.3.131.131" port: 45593 } }
I20260621 05:10:23.378324 5219 tablet_bootstrap.cc:654] T 2a9c9cc782ee48bc8c64b9fb93db7ea0 P b9643924b7a2495cbae396cd2b2523b4: Neither blocks nor log segments found. Creating new log.
I20260621 05:10:23.378350 5220 ts_tablet_manager.cc:1428] T 2a9c9cc782ee48bc8c64b9fb93db7ea0 P 98aae82ed2bd426fb5bd4fd0a69ede6f: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260621 05:10:23.378398 5221 raft_consensus.cc:348] T 2a9c9cc782ee48bc8c64b9fb93db7ea0 P 4f7fee4354ab4ed5b1bfae3ec9f66d4a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f7fee4354ab4ed5b1bfae3ec9f66d4a" member_type: VOTER last_known_addr { host: "127.3.131.129" port: 41195 } } peers { permanent_uuid: "98aae82ed2bd426fb5bd4fd0a69ede6f" member_type: VOTER last_known_addr { host: "127.3.131.130" port: 44133 } } peers { permanent_uuid: "b9643924b7a2495cbae396cd2b2523b4" member_type: VOTER last_known_addr { host: "127.3.131.131" port: 45593 } }
I20260621 05:10:23.378387 5237 tablet_replica.cc:439] T 2a9c9cc782ee48bc8c64b9fb93db7ea0 P 98aae82ed2bd426fb5bd4fd0a69ede6f: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f7fee4354ab4ed5b1bfae3ec9f66d4a" member_type: VOTER last_known_addr { host: "127.3.131.129" port: 41195 } } peers { permanent_uuid: "98aae82ed2bd426fb5bd4fd0a69ede6f" member_type: VOTER last_known_addr { host: "127.3.131.130" port: 44133 } } peers { permanent_uuid: "b9643924b7a2495cbae396cd2b2523b4" member_type: VOTER last_known_addr { host: "127.3.131.131" port: 45593 } } }
I20260621 05:10:23.378475 5221 raft_consensus.cc:374] T 2a9c9cc782ee48bc8c64b9fb93db7ea0 P 4f7fee4354ab4ed5b1bfae3ec9f66d4a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 05:10:23.378494 5221 raft_consensus.cc:729] T 2a9c9cc782ee48bc8c64b9fb93db7ea0 P 4f7fee4354ab4ed5b1bfae3ec9f66d4a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4f7fee4354ab4ed5b1bfae3ec9f66d4a, State: Initialized, Role: FOLLOWER
I20260621 05:10:23.378491 5237 tablet_replica.cc:442] T 2a9c9cc782ee48bc8c64b9fb93db7ea0 P 98aae82ed2bd426fb5bd4fd0a69ede6f: This TxnStatusTablet replica's current role is: FOLLOWER
I20260621 05:10:23.378532 5221 consensus_queue.cc:260] T 2a9c9cc782ee48bc8c64b9fb93db7ea0 P 4f7fee4354ab4ed5b1bfae3ec9f66d4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f7fee4354ab4ed5b1bfae3ec9f66d4a" member_type: VOTER last_known_addr { host: "127.3.131.129" port: 41195 } } peers { permanent_uuid: "98aae82ed2bd426fb5bd4fd0a69ede6f" member_type: VOTER last_known_addr { host: "127.3.131.130" port: 44133 } } peers { permanent_uuid: "b9643924b7a2495cbae396cd2b2523b4" member_type: VOTER last_known_addr { host: "127.3.131.131" port: 45593 } }
I20260621 05:10:23.378615 5221 ts_tablet_manager.cc:1428] T 2a9c9cc782ee48bc8c64b9fb93db7ea0 P 4f7fee4354ab4ed5b1bfae3ec9f66d4a: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260621 05:10:23.378669 5225 tablet_replica.cc:439] T 2a9c9cc782ee48bc8c64b9fb93db7ea0 P 4f7fee4354ab4ed5b1bfae3ec9f66d4a: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f7fee4354ab4ed5b1bfae3ec9f66d4a" member_type: VOTER last_known_addr { host: "127.3.131.129" port: 41195 } } peers { permanent_uuid: "98aae82ed2bd426fb5bd4fd0a69ede6f" member_type: VOTER last_known_addr { host: "127.3.131.130" port: 44133 } } peers { permanent_uuid: "b9643924b7a2495cbae396cd2b2523b4" member_type: VOTER last_known_addr { host: "127.3.131.131" port: 45593 } } }
I20260621 05:10:23.378729 5225 tablet_replica.cc:442] T 2a9c9cc782ee48bc8c64b9fb93db7ea0 P 4f7fee4354ab4ed5b1bfae3ec9f66d4a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260621 05:10:23.379088 5219 tablet_bootstrap.cc:492] T 2a9c9cc782ee48bc8c64b9fb93db7ea0 P b9643924b7a2495cbae396cd2b2523b4: No bootstrap required, opened a new log
I20260621 05:10:23.379142 5219 ts_tablet_manager.cc:1397] T 2a9c9cc782ee48bc8c64b9fb93db7ea0 P b9643924b7a2495cbae396cd2b2523b4: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260621 05:10:23.379456 5219 raft_consensus.cc:348] T 2a9c9cc782ee48bc8c64b9fb93db7ea0 P b9643924b7a2495cbae396cd2b2523b4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f7fee4354ab4ed5b1bfae3ec9f66d4a" member_type: VOTER last_known_addr { host: "127.3.131.129" port: 41195 } } peers { permanent_uuid: "98aae82ed2bd426fb5bd4fd0a69ede6f" member_type: VOTER last_known_addr { host: "127.3.131.130" port: 44133 } } peers { permanent_uuid: "b9643924b7a2495cbae396cd2b2523b4" member_type: VOTER last_known_addr { host: "127.3.131.131" port: 45593 } }
I20260621 05:10:23.379540 5219 raft_consensus.cc:374] T 2a9c9cc782ee48bc8c64b9fb93db7ea0 P b9643924b7a2495cbae396cd2b2523b4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 05:10:23.379563 5219 raft_consensus.cc:729] T 2a9c9cc782ee48bc8c64b9fb93db7ea0 P b9643924b7a2495cbae396cd2b2523b4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b9643924b7a2495cbae396cd2b2523b4, State: Initialized, Role: FOLLOWER
I20260621 05:10:23.379616 5219 consensus_queue.cc:260] T 2a9c9cc782ee48bc8c64b9fb93db7ea0 P b9643924b7a2495cbae396cd2b2523b4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f7fee4354ab4ed5b1bfae3ec9f66d4a" member_type: VOTER last_known_addr { host: "127.3.131.129" port: 41195 } } peers { permanent_uuid: "98aae82ed2bd426fb5bd4fd0a69ede6f" member_type: VOTER last_known_addr { host: "127.3.131.130" port: 44133 } } peers { permanent_uuid: "b9643924b7a2495cbae396cd2b2523b4" member_type: VOTER last_known_addr { host: "127.3.131.131" port: 45593 } }
I20260621 05:10:23.379725 5219 ts_tablet_manager.cc:1428] T 2a9c9cc782ee48bc8c64b9fb93db7ea0 P b9643924b7a2495cbae396cd2b2523b4: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260621 05:10:23.379733 5226 tablet_replica.cc:439] T 2a9c9cc782ee48bc8c64b9fb93db7ea0 P b9643924b7a2495cbae396cd2b2523b4: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f7fee4354ab4ed5b1bfae3ec9f66d4a" member_type: VOTER last_known_addr { host: "127.3.131.129" port: 41195 } } peers { permanent_uuid: "98aae82ed2bd426fb5bd4fd0a69ede6f" member_type: VOTER last_known_addr { host: "127.3.131.130" port: 44133 } } peers { permanent_uuid: "b9643924b7a2495cbae396cd2b2523b4" member_type: VOTER last_known_addr { host: "127.3.131.131" port: 45593 } } }
I20260621 05:10:23.379861 5226 tablet_replica.cc:442] T 2a9c9cc782ee48bc8c64b9fb93db7ea0 P b9643924b7a2495cbae396cd2b2523b4: This TxnStatusTablet replica's current role is: FOLLOWER
I20260621 05:10:23.572352 5237 raft_consensus.cc:482] T 2a9c9cc782ee48bc8c64b9fb93db7ea0 P 98aae82ed2bd426fb5bd4fd0a69ede6f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260621 05:10:23.572479 5237 raft_consensus.cc:504] T 2a9c9cc782ee48bc8c64b9fb93db7ea0 P 98aae82ed2bd426fb5bd4fd0a69ede6f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f7fee4354ab4ed5b1bfae3ec9f66d4a" member_type: VOTER last_known_addr { host: "127.3.131.129" port: 41195 } } peers { permanent_uuid: "98aae82ed2bd426fb5bd4fd0a69ede6f" member_type: VOTER last_known_addr { host: "127.3.131.130" port: 44133 } } peers { permanent_uuid: "b9643924b7a2495cbae396cd2b2523b4" member_type: VOTER last_known_addr { host: "127.3.131.131" port: 45593 } }
I20260621 05:10:23.572726 5237 leader_election.cc:288] T 2a9c9cc782ee48bc8c64b9fb93db7ea0 P 98aae82ed2bd426fb5bd4fd0a69ede6f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4f7fee4354ab4ed5b1bfae3ec9f66d4a (127.3.131.129:41195), b9643924b7a2495cbae396cd2b2523b4 (127.3.131.131:45593)
I20260621 05:10:23.573094 4870 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a9c9cc782ee48bc8c64b9fb93db7ea0" candidate_uuid: "98aae82ed2bd426fb5bd4fd0a69ede6f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4f7fee4354ab4ed5b1bfae3ec9f66d4a" is_pre_election: true
I20260621 05:10:23.573093 5150 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a9c9cc782ee48bc8c64b9fb93db7ea0" candidate_uuid: "98aae82ed2bd426fb5bd4fd0a69ede6f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9643924b7a2495cbae396cd2b2523b4" is_pre_election: true
I20260621 05:10:23.573192 4870 raft_consensus.cc:2454] T 2a9c9cc782ee48bc8c64b9fb93db7ea0 P 4f7fee4354ab4ed5b1bfae3ec9f66d4a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 98aae82ed2bd426fb5bd4fd0a69ede6f in term 0.
I20260621 05:10:23.573212 5150 raft_consensus.cc:2454] T 2a9c9cc782ee48bc8c64b9fb93db7ea0 P b9643924b7a2495cbae396cd2b2523b4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 98aae82ed2bd426fb5bd4fd0a69ede6f in term 0.
I20260621 05:10:23.573377 4941 leader_election.cc:302] T 2a9c9cc782ee48bc8c64b9fb93db7ea0 P 98aae82ed2bd426fb5bd4fd0a69ede6f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 98aae82ed2bd426fb5bd4fd0a69ede6f, b9643924b7a2495cbae396cd2b2523b4; no voters:
I20260621 05:10:23.573551 5237 raft_consensus.cc:2781] T 2a9c9cc782ee48bc8c64b9fb93db7ea0 P 98aae82ed2bd426fb5bd4fd0a69ede6f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260621 05:10:23.573606 5237 raft_consensus.cc:482] T 2a9c9cc782ee48bc8c64b9fb93db7ea0 P 98aae82ed2bd426fb5bd4fd0a69ede6f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260621 05:10:23.573630 5237 raft_consensus.cc:3037] T 2a9c9cc782ee48bc8c64b9fb93db7ea0 P 98aae82ed2bd426fb5bd4fd0a69ede6f [term 0 FOLLOWER]: Advancing to term 1
I20260621 05:10:23.574414 5237 raft_consensus.cc:504] T 2a9c9cc782ee48bc8c64b9fb93db7ea0 P 98aae82ed2bd426fb5bd4fd0a69ede6f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f7fee4354ab4ed5b1bfae3ec9f66d4a" member_type: VOTER last_known_addr { host: "127.3.131.129" port: 41195 } } peers { permanent_uuid: "98aae82ed2bd426fb5bd4fd0a69ede6f" member_type: VOTER last_known_addr { host: "127.3.131.130" port: 44133 } } peers { permanent_uuid: "b9643924b7a2495cbae396cd2b2523b4" member_type: VOTER last_known_addr { host: "127.3.131.131" port: 45593 } }
I20260621 05:10:23.574566 5237 leader_election.cc:288] T 2a9c9cc782ee48bc8c64b9fb93db7ea0 P 98aae82ed2bd426fb5bd4fd0a69ede6f [CANDIDATE]: Term 1 election: Requested vote from peers 4f7fee4354ab4ed5b1bfae3ec9f66d4a (127.3.131.129:41195), b9643924b7a2495cbae396cd2b2523b4 (127.3.131.131:45593)
I20260621 05:10:23.574771 4870 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a9c9cc782ee48bc8c64b9fb93db7ea0" candidate_uuid: "98aae82ed2bd426fb5bd4fd0a69ede6f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4f7fee4354ab4ed5b1bfae3ec9f66d4a"
I20260621 05:10:23.574787 5150 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a9c9cc782ee48bc8c64b9fb93db7ea0" candidate_uuid: "98aae82ed2bd426fb5bd4fd0a69ede6f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9643924b7a2495cbae396cd2b2523b4"
I20260621 05:10:23.574879 5150 raft_consensus.cc:3037] T 2a9c9cc782ee48bc8c64b9fb93db7ea0 P b9643924b7a2495cbae396cd2b2523b4 [term 0 FOLLOWER]: Advancing to term 1
I20260621 05:10:23.574882 4870 raft_consensus.cc:3037] T 2a9c9cc782ee48bc8c64b9fb93db7ea0 P 4f7fee4354ab4ed5b1bfae3ec9f66d4a [term 0 FOLLOWER]: Advancing to term 1
I20260621 05:10:23.575618 5150 raft_consensus.cc:2454] T 2a9c9cc782ee48bc8c64b9fb93db7ea0 P b9643924b7a2495cbae396cd2b2523b4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 98aae82ed2bd426fb5bd4fd0a69ede6f in term 1.
I20260621 05:10:23.575706 4870 raft_consensus.cc:2454] T 2a9c9cc782ee48bc8c64b9fb93db7ea0 P 4f7fee4354ab4ed5b1bfae3ec9f66d4a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 98aae82ed2bd426fb5bd4fd0a69ede6f in term 1.
I20260621 05:10:23.575814 4941 leader_election.cc:302] T 2a9c9cc782ee48bc8c64b9fb93db7ea0 P 98aae82ed2bd426fb5bd4fd0a69ede6f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 98aae82ed2bd426fb5bd4fd0a69ede6f, b9643924b7a2495cbae396cd2b2523b4; no voters:
I20260621 05:10:23.575935 5237 raft_consensus.cc:2781] T 2a9c9cc782ee48bc8c64b9fb93db7ea0 P 98aae82ed2bd426fb5bd4fd0a69ede6f [term 1 FOLLOWER]: Leader election won for term 1
I20260621 05:10:23.576015 5237 raft_consensus.cc:686] T 2a9c9cc782ee48bc8c64b9fb93db7ea0 P 98aae82ed2bd426fb5bd4fd0a69ede6f [term 1 LEADER]: Becoming Leader. State: Replica: 98aae82ed2bd426fb5bd4fd0a69ede6f, State: Running, Role: LEADER
I20260621 05:10:23.576077 5237 consensus_queue.cc:237] T 2a9c9cc782ee48bc8c64b9fb93db7ea0 P 98aae82ed2bd426fb5bd4fd0a69ede6f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f7fee4354ab4ed5b1bfae3ec9f66d4a" member_type: VOTER last_known_addr { host: "127.3.131.129" port: 41195 } } peers { permanent_uuid: "98aae82ed2bd426fb5bd4fd0a69ede6f" member_type: VOTER last_known_addr { host: "127.3.131.130" port: 44133 } } peers { permanent_uuid: "b9643924b7a2495cbae396cd2b2523b4" member_type: VOTER last_known_addr { host: "127.3.131.131" port: 45593 } }
I20260621 05:10:23.576387 5229 tablet_replica.cc:439] T 2a9c9cc782ee48bc8c64b9fb93db7ea0 P 98aae82ed2bd426fb5bd4fd0a69ede6f: TxnStatusTablet state changed. Reason: New leader 98aae82ed2bd426fb5bd4fd0a69ede6f. Latest consensus state: current_term: 1 leader_uuid: "98aae82ed2bd426fb5bd4fd0a69ede6f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f7fee4354ab4ed5b1bfae3ec9f66d4a" member_type: VOTER last_known_addr { host: "127.3.131.129" port: 41195 } } peers { permanent_uuid: "98aae82ed2bd426fb5bd4fd0a69ede6f" member_type: VOTER last_known_addr { host: "127.3.131.130" port: 44133 } } peers { permanent_uuid: "b9643924b7a2495cbae396cd2b2523b4" member_type: VOTER last_known_addr { host: "127.3.131.131" port: 45593 } } }
I20260621 05:10:23.576480 5229 tablet_replica.cc:442] T 2a9c9cc782ee48bc8c64b9fb93db7ea0 P 98aae82ed2bd426fb5bd4fd0a69ede6f: This TxnStatusTablet replica's current role is: LEADER
I20260621 05:10:23.576745 5261 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260621 05:10:23.576884 5261 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260621 05:10:23.576792 4711 catalog_manager.cc:5495] T 2a9c9cc782ee48bc8c64b9fb93db7ea0 P 98aae82ed2bd426fb5bd4fd0a69ede6f reported cstate change: term changed from 0 to 1, leader changed from <none> to 98aae82ed2bd426fb5bd4fd0a69ede6f (127.3.131.130). New cstate: current_term: 1 leader_uuid: "98aae82ed2bd426fb5bd4fd0a69ede6f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f7fee4354ab4ed5b1bfae3ec9f66d4a" member_type: VOTER last_known_addr { host: "127.3.131.129" port: 41195 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "98aae82ed2bd426fb5bd4fd0a69ede6f" member_type: VOTER last_known_addr { host: "127.3.131.130" port: 44133 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b9643924b7a2495cbae396cd2b2523b4" member_type: VOTER last_known_addr { host: "127.3.131.131" port: 45593 } health_report { overall_health: UNKNOWN } } }
I20260621 05:10:23.609804 5217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:10:23.607441 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58828 (local address 127.3.131.130:44133)
0621 05:10:23.607487 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0621 05:10:23.607489 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:10:23.607496 (+ 7us) server_negotiation.cc:407] Connection header received
0621 05:10:23.607518 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:10:23.607519 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:10:23.607528 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:10:23.607592 (+ 64us) server_negotiation.cc:226] Negotiated authn=TOKEN
0621 05:10:23.607758 (+ 166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:10:23.608251 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:10:23.609041 (+ 790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:10:23.609241 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:10:23.609319 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:10:23.609518 (+ 199us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0621 05:10:23.609560 (+ 42us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0621 05:10:23.609621 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0621 05:10:23.609696 (+ 75us) server_negotiation.cc:299] Negotiation successful
0621 05:10:23.609720 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260621 05:10:23.609818 5256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:10:23.607236 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.131.130:44133 (local address 127.0.0.1:58828)
0621 05:10:23.607315 (+ 79us) negotiation.cc:107] Waiting for socket to connect
0621 05:10:23.607325 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0621 05:10:23.607374 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:10:23.607618 (+ 244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:10:23.607619 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:10:23.607620 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0621 05:10:23.607675 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:10:23.607681 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:10:23.608338 (+ 657us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:10:23.608340 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:10:23.608947 (+ 607us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:10:23.608950 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:10:23.609318 (+ 368us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:10:23.609319 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:10:23.609400 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:10:23.609418 (+ 18us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0621 05:10:23.609623 (+ 205us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0621 05:10:23.609634 (+ 11us) client_negotiation.cc:769] Sending connection context
0621 05:10:23.609690 (+ 56us) client_negotiation.cc:240] Negotiation successful
0621 05:10:23.609749 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":27}
I20260621 05:10:23.611511 5150 raft_consensus.cc:1261] T 2a9c9cc782ee48bc8c64b9fb93db7ea0 P b9643924b7a2495cbae396cd2b2523b4 [term 1 FOLLOWER]: Refusing update from remote peer 98aae82ed2bd426fb5bd4fd0a69ede6f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 05:10:23.611542 4870 raft_consensus.cc:1261] T 2a9c9cc782ee48bc8c64b9fb93db7ea0 P 4f7fee4354ab4ed5b1bfae3ec9f66d4a [term 1 FOLLOWER]: Refusing update from remote peer 98aae82ed2bd426fb5bd4fd0a69ede6f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 05:10:23.611783 5237 consensus_queue.cc:1035] T 2a9c9cc782ee48bc8c64b9fb93db7ea0 P 98aae82ed2bd426fb5bd4fd0a69ede6f [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9643924b7a2495cbae396cd2b2523b4" member_type: VOTER last_known_addr { host: "127.3.131.131" port: 45593 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 05:10:23.611917 5229 consensus_queue.cc:1035] T 2a9c9cc782ee48bc8c64b9fb93db7ea0 P 98aae82ed2bd426fb5bd4fd0a69ede6f [LEADER]: Connected to new peer: Peer: permanent_uuid: "4f7fee4354ab4ed5b1bfae3ec9f66d4a" member_type: VOTER last_known_addr { host: "127.3.131.129" port: 41195 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 05:10:23.612694 5225 tablet_replica.cc:439] T 2a9c9cc782ee48bc8c64b9fb93db7ea0 P 4f7fee4354ab4ed5b1bfae3ec9f66d4a: TxnStatusTablet state changed. Reason: New leader 98aae82ed2bd426fb5bd4fd0a69ede6f. Latest consensus state: current_term: 1 leader_uuid: "98aae82ed2bd426fb5bd4fd0a69ede6f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f7fee4354ab4ed5b1bfae3ec9f66d4a" member_type: VOTER last_known_addr { host: "127.3.131.129" port: 41195 } } peers { permanent_uuid: "98aae82ed2bd426fb5bd4fd0a69ede6f" member_type: VOTER last_known_addr { host: "127.3.131.130" port: 44133 } } peers { permanent_uuid: "b9643924b7a2495cbae396cd2b2523b4" member_type: VOTER last_known_addr { host: "127.3.131.131" port: 45593 } } }
I20260621 05:10:23.612808 5225 tablet_replica.cc:442] T 2a9c9cc782ee48bc8c64b9fb93db7ea0 P 4f7fee4354ab4ed5b1bfae3ec9f66d4a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260621 05:10:23.612897 5226 tablet_replica.cc:439] T 2a9c9cc782ee48bc8c64b9fb93db7ea0 P b9643924b7a2495cbae396cd2b2523b4: TxnStatusTablet state changed. Reason: New leader 98aae82ed2bd426fb5bd4fd0a69ede6f. Latest consensus state: current_term: 1 leader_uuid: "98aae82ed2bd426fb5bd4fd0a69ede6f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f7fee4354ab4ed5b1bfae3ec9f66d4a" member_type: VOTER last_known_addr { host: "127.3.131.129" port: 41195 } } peers { permanent_uuid: "98aae82ed2bd426fb5bd4fd0a69ede6f" member_type: VOTER last_known_addr { host: "127.3.131.130" port: 44133 } } peers { permanent_uuid: "b9643924b7a2495cbae396cd2b2523b4" member_type: VOTER last_known_addr { host: "127.3.131.131" port: 45593 } } }
I20260621 05:10:23.613197 5226 tablet_replica.cc:442] T 2a9c9cc782ee48bc8c64b9fb93db7ea0 P b9643924b7a2495cbae396cd2b2523b4: This TxnStatusTablet replica's current role is: FOLLOWER
I20260621 05:10:23.613298 5237 tablet_replica.cc:439] T 2a9c9cc782ee48bc8c64b9fb93db7ea0 P 98aae82ed2bd426fb5bd4fd0a69ede6f: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "98aae82ed2bd426fb5bd4fd0a69ede6f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f7fee4354ab4ed5b1bfae3ec9f66d4a" member_type: VOTER last_known_addr { host: "127.3.131.129" port: 41195 } } peers { permanent_uuid: "98aae82ed2bd426fb5bd4fd0a69ede6f" member_type: VOTER last_known_addr { host: "127.3.131.130" port: 44133 } } peers { permanent_uuid: "b9643924b7a2495cbae396cd2b2523b4" member_type: VOTER last_known_addr { host: "127.3.131.131" port: 45593 } } }
I20260621 05:10:23.613437 5237 tablet_replica.cc:442] T 2a9c9cc782ee48bc8c64b9fb93db7ea0 P 98aae82ed2bd426fb5bd4fd0a69ede6f: This TxnStatusTablet replica's current role is: LEADER
I20260621 05:10:23.614169 5237 tablet_replica.cc:439] T 2a9c9cc782ee48bc8c64b9fb93db7ea0 P 98aae82ed2bd426fb5bd4fd0a69ede6f: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "98aae82ed2bd426fb5bd4fd0a69ede6f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f7fee4354ab4ed5b1bfae3ec9f66d4a" member_type: VOTER last_known_addr { host: "127.3.131.129" port: 41195 } } peers { permanent_uuid: "98aae82ed2bd426fb5bd4fd0a69ede6f" member_type: VOTER last_known_addr { host: "127.3.131.130" port: 44133 } } peers { permanent_uuid: "b9643924b7a2495cbae396cd2b2523b4" member_type: VOTER last_known_addr { host: "127.3.131.131" port: 45593 } } }
I20260621 05:10:23.614264 5237 tablet_replica.cc:442] T 2a9c9cc782ee48bc8c64b9fb93db7ea0 P 98aae82ed2bd426fb5bd4fd0a69ede6f: This TxnStatusTablet replica's current role is: LEADER
I20260621 05:10:23.614259 5225 tablet_replica.cc:439] T 2a9c9cc782ee48bc8c64b9fb93db7ea0 P 4f7fee4354ab4ed5b1bfae3ec9f66d4a: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "98aae82ed2bd426fb5bd4fd0a69ede6f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f7fee4354ab4ed5b1bfae3ec9f66d4a" member_type: VOTER last_known_addr { host: "127.3.131.129" port: 41195 } } peers { permanent_uuid: "98aae82ed2bd426fb5bd4fd0a69ede6f" member_type: VOTER last_known_addr { host: "127.3.131.130" port: 44133 } } peers { permanent_uuid: "b9643924b7a2495cbae396cd2b2523b4" member_type: VOTER last_known_addr { host: "127.3.131.131" port: 45593 } } }
I20260621 05:10:23.614351 5225 tablet_replica.cc:442] T 2a9c9cc782ee48bc8c64b9fb93db7ea0 P 4f7fee4354ab4ed5b1bfae3ec9f66d4a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260621 05:10:23.614943 5226 tablet_replica.cc:439] T 2a9c9cc782ee48bc8c64b9fb93db7ea0 P b9643924b7a2495cbae396cd2b2523b4: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "98aae82ed2bd426fb5bd4fd0a69ede6f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f7fee4354ab4ed5b1bfae3ec9f66d4a" member_type: VOTER last_known_addr { host: "127.3.131.129" port: 41195 } } peers { permanent_uuid: "98aae82ed2bd426fb5bd4fd0a69ede6f" member_type: VOTER last_known_addr { host: "127.3.131.130" port: 44133 } } peers { permanent_uuid: "b9643924b7a2495cbae396cd2b2523b4" member_type: VOTER last_known_addr { host: "127.3.131.131" port: 45593 } } }
I20260621 05:10:23.615041 5226 tablet_replica.cc:442] T 2a9c9cc782ee48bc8c64b9fb93db7ea0 P b9643924b7a2495cbae396cd2b2523b4: This TxnStatusTablet replica's current role is: FOLLOWER
I20260621 05:10:23.619452 5274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:10:23.616656 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54094 (local address 127.3.131.131:45593)
0621 05:10:23.616799 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0621 05:10:23.616801 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:10:23.616808 (+ 7us) server_negotiation.cc:407] Connection header received
0621 05:10:23.616863 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:10:23.616864 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:10:23.616884 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:10:23.616976 (+ 92us) server_negotiation.cc:226] Negotiated authn=TOKEN
0621 05:10:23.617113 (+ 137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:10:23.617652 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:10:23.618435 (+ 783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:10:23.618678 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:10:23.618757 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:10:23.618942 (+ 185us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0621 05:10:23.619067 (+ 125us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0621 05:10:23.619149 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0621 05:10:23.619260 (+ 111us) server_negotiation.cc:299] Negotiation successful
0621 05:10:23.619350 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":124,"thread_start_us":82,"threads_started":1}
I20260621 05:10:23.627015 5277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:10:23.622656 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.131.130:44133 (local address 127.3.131.131:52173)
0621 05:10:23.622845 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0621 05:10:23.622855 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0621 05:10:23.622911 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:10:23.623121 (+ 210us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:10:23.623122 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:10:23.623124 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0621 05:10:23.623304 (+ 180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:10:23.623307 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:10:23.624605 (+ 1298us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:10:23.624606 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:10:23.625532 (+ 926us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:10:23.625537 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:10:23.626383 (+ 846us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:10:23.626385 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:10:23.626446 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:10:23.626468 (+ 22us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0621 05:10:23.626765 (+ 297us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0621 05:10:23.626768 (+ 3us) client_negotiation.cc:769] Sending connection context
0621 05:10:23.626883 (+ 115us) client_negotiation.cc:240] Negotiation successful
0621 05:10:23.626930 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":97,"threads_started":1}
I20260621 05:10:23.627025 5217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:10:23.622732 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.131.131:52173 (local address 127.3.131.130:44133)
0621 05:10:23.622813 (+ 81us) server_negotiation.cc:206] Beginning negotiation
0621 05:10:23.622815 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:10:23.622918 (+ 103us) server_negotiation.cc:407] Connection header received
0621 05:10:23.622967 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:10:23.622968 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:10:23.622982 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:10:23.623042 (+ 60us) server_negotiation.cc:226] Negotiated authn=TOKEN
0621 05:10:23.623464 (+ 422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:10:23.624419 (+ 955us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:10:23.625660 (+ 1241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:10:23.625974 (+ 314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:10:23.626060 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:10:23.626613 (+ 553us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0621 05:10:23.626656 (+ 43us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0621 05:10:23.626729 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0621 05:10:23.626899 (+ 170us) server_negotiation.cc:299] Negotiation successful
0621 05:10:23.626928 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260621 05:10:23.630924 4870 raft_consensus.cc:1261] T 1440e0f2d5aa4770a628324cebdb279d P 4f7fee4354ab4ed5b1bfae3ec9f66d4a [term 1 FOLLOWER]: Refusing update from remote peer b9643924b7a2495cbae396cd2b2523b4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 05:10:23.630998 5010 raft_consensus.cc:1261] T 1440e0f2d5aa4770a628324cebdb279d P 98aae82ed2bd426fb5bd4fd0a69ede6f [term 1 FOLLOWER]: Refusing update from remote peer b9643924b7a2495cbae396cd2b2523b4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 05:10:23.631384 5249 consensus_queue.cc:1035] T 1440e0f2d5aa4770a628324cebdb279d P b9643924b7a2495cbae396cd2b2523b4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4f7fee4354ab4ed5b1bfae3ec9f66d4a" member_type: VOTER last_known_addr { host: "127.3.131.129" port: 41195 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 05:10:23.631567 5226 consensus_queue.cc:1035] T 1440e0f2d5aa4770a628324cebdb279d P b9643924b7a2495cbae396cd2b2523b4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "98aae82ed2bd426fb5bd4fd0a69ede6f" member_type: VOTER last_known_addr { host: "127.3.131.130" port: 44133 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 05:10:23.636727 5264 mvcc.cc:205] Tried to move back new op lower bound from 7299148282389389312 to 7299148281066967040. Current Snapshot: MvccSnapshot[applied={T|T < 7299148282389389312}]
I20260621 05:10:23.641745 5282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:10:23.638366 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.131.131:45593 (local address 127.3.131.130:37421)
0621 05:10:23.638534 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0621 05:10:23.638546 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0621 05:10:23.638616 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:10:23.638759 (+ 143us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:10:23.638760 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:10:23.638762 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0621 05:10:23.638820 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:10:23.638821 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:10:23.639788 (+ 967us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:10:23.639789 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:10:23.640617 (+ 828us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:10:23.640620 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:10:23.641195 (+ 575us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:10:23.641196 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:10:23.641246 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:10:23.641261 (+ 15us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0621 05:10:23.641504 (+ 243us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0621 05:10:23.641506 (+ 2us) client_negotiation.cc:769] Sending connection context
0621 05:10:23.641590 (+ 84us) client_negotiation.cc:240] Negotiation successful
0621 05:10:23.641644 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":90,"threads_started":1}
I20260621 05:10:23.641737 5274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:10:23.638415 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.131.130:37421 (local address 127.3.131.131:45593)
0621 05:10:23.638456 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0621 05:10:23.638458 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:10:23.638626 (+ 168us) server_negotiation.cc:407] Connection header received
0621 05:10:23.638682 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:10:23.638682 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:10:23.638693 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:10:23.638751 (+ 58us) server_negotiation.cc:226] Negotiated authn=TOKEN
0621 05:10:23.638893 (+ 142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:10:23.639674 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:10:23.640755 (+ 1081us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:10:23.641108 (+ 353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:10:23.641187 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:10:23.641407 (+ 220us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0621 05:10:23.641444 (+ 37us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0621 05:10:23.641497 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0621 05:10:23.641596 (+ 99us) server_negotiation.cc:299] Negotiation successful
0621 05:10:23.641619 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260621 05:10:23.658524 4712 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46762:
table { table_name: "test-table" } modify_external_catalogs: true
I20260621 05:10:23.658668 4712 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46762:
table { table_name: "test-table" } modify_external_catalogs: true
I20260621 05:10:23.659533 4712 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d11704c4279f426fa489c4af98674cd5: Sending DeleteTablet for 3 replicas of tablet 1440e0f2d5aa4770a628324cebdb279d
I20260621 05:10:23.660010 4850 tablet_service.cc:1514] Processing DeleteTablet for tablet 1440e0f2d5aa4770a628324cebdb279d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 05:10:23 UTC) from {username='kudu', principal='kudu/127.3.131.190@KRBTEST.COM'} at 127.0.0.1:51878
I20260621 05:10:23.660387 5285 tablet_replica.cc:330] T 1440e0f2d5aa4770a628324cebdb279d P 4f7fee4354ab4ed5b1bfae3ec9f66d4a: stopping tablet replica
I20260621 05:10:23.660390 5130 tablet_service.cc:1514] Processing DeleteTablet for tablet 1440e0f2d5aa4770a628324cebdb279d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 05:10:23 UTC) from {username='kudu', principal='kudu/127.3.131.190@KRBTEST.COM'} at 127.0.0.1:54090
I20260621 05:10:23.660522 5285 raft_consensus.cc:2229] T 1440e0f2d5aa4770a628324cebdb279d P 4f7fee4354ab4ed5b1bfae3ec9f66d4a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260621 05:10:23.660588 5285 raft_consensus.cc:2258] T 1440e0f2d5aa4770a628324cebdb279d P 4f7fee4354ab4ed5b1bfae3ec9f66d4a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 05:10:23.660904 5285 ts_tablet_manager.cc:1905] T 1440e0f2d5aa4770a628324cebdb279d P 4f7fee4354ab4ed5b1bfae3ec9f66d4a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 05:10:23.661015 4990 tablet_service.cc:1514] Processing DeleteTablet for tablet 1440e0f2d5aa4770a628324cebdb279d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 05:10:23 UTC) from {username='kudu', principal='kudu/127.3.131.190@KRBTEST.COM'} at 127.0.0.1:58808
I20260621 05:10:23.661950 5287 tablet_replica.cc:330] T 1440e0f2d5aa4770a628324cebdb279d P 98aae82ed2bd426fb5bd4fd0a69ede6f: stopping tablet replica
I20260621 05:10:23.662099 5287 raft_consensus.cc:2229] T 1440e0f2d5aa4770a628324cebdb279d P 98aae82ed2bd426fb5bd4fd0a69ede6f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260621 05:10:23.662209 5287 raft_consensus.cc:2258] T 1440e0f2d5aa4770a628324cebdb279d P 98aae82ed2bd426fb5bd4fd0a69ede6f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 05:10:23.662590 5287 ts_tablet_manager.cc:1905] T 1440e0f2d5aa4770a628324cebdb279d P 98aae82ed2bd426fb5bd4fd0a69ede6f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 05:10:23.663964 5285 ts_tablet_manager.cc:1918] T 1440e0f2d5aa4770a628324cebdb279d P 4f7fee4354ab4ed5b1bfae3ec9f66d4a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260621 05:10:23.664083 5285 log.cc:1196] T 1440e0f2d5aa4770a628324cebdb279d P 4f7fee4354ab4ed5b1bfae3ec9f66d4a: Deleting WAL directory at /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782018616648578-3598-0/minicluster-data/ts-0/wal/wals/1440e0f2d5aa4770a628324cebdb279d
I20260621 05:10:23.664311 5286 tablet_replica.cc:330] T 1440e0f2d5aa4770a628324cebdb279d P b9643924b7a2495cbae396cd2b2523b4: stopping tablet replica
I20260621 05:10:23.664418 5285 ts_tablet_manager.cc:1939] T 1440e0f2d5aa4770a628324cebdb279d P 4f7fee4354ab4ed5b1bfae3ec9f66d4a: Deleting consensus metadata
I20260621 05:10:23.664844 5286 raft_consensus.cc:2229] T 1440e0f2d5aa4770a628324cebdb279d P b9643924b7a2495cbae396cd2b2523b4 [term 1 LEADER]: Raft consensus shutting down.
I20260621 05:10:23.664985 5286 raft_consensus.cc:2258] T 1440e0f2d5aa4770a628324cebdb279d P b9643924b7a2495cbae396cd2b2523b4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 05:10:23.665112 4697 catalog_manager.cc:4841] TS 4f7fee4354ab4ed5b1bfae3ec9f66d4a (127.3.131.129:41195): tablet 1440e0f2d5aa4770a628324cebdb279d (table test-table [id=141d3bfb2af14566b40b4f7a3f685a03]) successfully deleted
I20260621 05:10:23.665333 5286 ts_tablet_manager.cc:1905] T 1440e0f2d5aa4770a628324cebdb279d P b9643924b7a2495cbae396cd2b2523b4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 05:10:23.667790 5286 ts_tablet_manager.cc:1918] T 1440e0f2d5aa4770a628324cebdb279d P b9643924b7a2495cbae396cd2b2523b4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260621 05:10:23.669411 5287 ts_tablet_manager.cc:1918] T 1440e0f2d5aa4770a628324cebdb279d P 98aae82ed2bd426fb5bd4fd0a69ede6f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260621 05:10:23.669764 5287 log.cc:1196] T 1440e0f2d5aa4770a628324cebdb279d P 98aae82ed2bd426fb5bd4fd0a69ede6f: Deleting WAL directory at /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782018616648578-3598-0/minicluster-data/ts-1/wal/wals/1440e0f2d5aa4770a628324cebdb279d
I20260621 05:10:23.670118 5287 ts_tablet_manager.cc:1939] T 1440e0f2d5aa4770a628324cebdb279d P 98aae82ed2bd426fb5bd4fd0a69ede6f: Deleting consensus metadata
I20260621 05:10:23.670569 4699 catalog_manager.cc:4841] TS 98aae82ed2bd426fb5bd4fd0a69ede6f (127.3.131.130:44133): tablet 1440e0f2d5aa4770a628324cebdb279d (table test-table [id=141d3bfb2af14566b40b4f7a3f685a03]) successfully deleted
I20260621 05:10:23.670830 5286 log.cc:1196] T 1440e0f2d5aa4770a628324cebdb279d P b9643924b7a2495cbae396cd2b2523b4: Deleting WAL directory at /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782018616648578-3598-0/minicluster-data/ts-2/wal/wals/1440e0f2d5aa4770a628324cebdb279d
I20260621 05:10:23.671209 5286 ts_tablet_manager.cc:1939] T 1440e0f2d5aa4770a628324cebdb279d P b9643924b7a2495cbae396cd2b2523b4: Deleting consensus metadata
I20260621 05:10:23.671784 4697 catalog_manager.cc:4841] TS b9643924b7a2495cbae396cd2b2523b4 (127.3.131.131:45593): tablet 1440e0f2d5aa4770a628324cebdb279d (table test-table [id=141d3bfb2af14566b40b4f7a3f685a03]) successfully deleted
Jun 21 05:10:23 dist-test-slave-vrq1 krb5kdc[4668](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782018622, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.3.131.129@KRBTEST.COM
I20260621 05:10:23.680568 5218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:10:23.664714 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51890 (local address 127.3.131.129:41195)
0621 05:10:23.664773 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0621 05:10:23.664775 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:10:23.667480 (+ 2705us) server_negotiation.cc:407] Connection header received
0621 05:10:23.667768 (+ 288us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:10:23.667770 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:10:23.667791 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:10:23.667830 (+ 39us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:10:23.669063 (+ 1233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:10:23.670057 (+ 994us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:10:23.671361 (+ 1304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:10:23.671766 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:10:23.671824 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:10:23.676233 (+ 4409us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:10:23.676249 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:10:23.676249 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:10:23.676262 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:10:23.678651 (+ 2389us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:10:23.679456 (+ 805us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:10:23.679461 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:10:23.679461 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:10:23.679545 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:10:23.679842 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:10:23.679843 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:10:23.679843 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:10:23.680059 (+ 216us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:10:23.680215 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0621 05:10:23.680355 (+ 140us) server_negotiation.cc:299] Negotiation successful
0621 05:10:23.680456 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
W20260621 05:10:23.680939 4906 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:51890
I20260621 05:10:23.690506 4763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:10:23.682049 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46790 (local address 127.3.131.190:43199)
0621 05:10:23.682114 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0621 05:10:23.682116 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:10:23.682271 (+ 155us) server_negotiation.cc:407] Connection header received
0621 05:10:23.682339 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:10:23.682340 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:10:23.682355 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:10:23.682424 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:10:23.683110 (+ 686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:10:23.684032 (+ 922us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:10:23.685206 (+ 1174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:10:23.685559 (+ 353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:10:23.685639 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:10:23.687349 (+ 1710us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:10:23.687351 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:10:23.687351 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:10:23.687360 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:10:23.689178 (+ 1818us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:10:23.689603 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:10:23.689604 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:10:23.689605 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:10:23.689675 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:10:23.689910 (+ 235us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:10:23.689911 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:10:23.689912 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:10:23.690067 (+ 155us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:10:23.690181 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0621 05:10:23.690310 (+ 129us) server_negotiation.cc:299] Negotiation successful
0621 05:10:23.690400 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
W20260621 05:10:23.690889 4712 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46790
I20260621 05:10:23.691496 3598 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWZOx2_/build/release/bin/kudu with pid 4785
I20260621 05:10:23.698179 3598 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWZOx2_/build/release/bin/kudu with pid 4925
I20260621 05:10:23.704078 3598 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWZOx2_/build/release/bin/kudu with pid 5065
I20260621 05:10:23.709107 3598 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWZOx2_/build/release/bin/kudu with pid 4681
2026-06-21T05:10:23Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (2609 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782018616648578-3598-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 21 05:10:23 dist-test-slave-vrq1 krb5kdc[5294](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 21 05:10:23 dist-test-slave-vrq1 krb5kdc[5294](info): set up 2 sockets
Jun 21 05:10:23 dist-test-slave-vrq1 krb5kdc[5294](info): commencing operation
krb5kdc: starting...
W20260621 05:10:24.775516 3598 mini_kdc.cc:121] Time spent starting KDC: real 1.044s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782018616648578-3598-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782018616648578-3598-0/krb5kdc/test-user.keytab.
Jun 21 05:10:24 dist-test-slave-vrq1 krb5kdc[5294](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782018624, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-21T05:10:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-21T05:10:24Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.131.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.131.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.131.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782018616648578-3598-0/krb5kdc/kudu_127.3.131.190.keytab.
Entry for principal kudu/127.3.131.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782018616648578-3598-0/krb5kdc/kudu_127.3.131.190.keytab.
I20260621 05:10:24.910467 3598 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWZOx2_/build/release/bin/kudu
/tmp/dist-test-taskWZOx2_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782018616648578-3598-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782018616648578-3598-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782018616648578-3598-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782018616648578-3598-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.131.190:43783
--webserver_interface=127.3.131.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782018616648578-3598-0/krb5kdc/kudu_127.3.131.190.keytab
--principal=kudu/127.3.131.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.131.148:35995
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.131.190:43783
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782018616648578-3598-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782018616648578-3598-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782018616648578-3598-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 05:10:24.991926 5307 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 05:10:24.992218 5307 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 05:10:24.992266 5307 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 05:10:24.993472 5307 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 05:10:24.993541 5307 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260621 05:10:24.993566 5307 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 05:10:24.993588 5307 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 05:10:24.993621 5307 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 05:10:24.993645 5307 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 05:10:24.995196 5307 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.131.148:35995
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782018616648578-3598-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782018616648578-3598-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.131.190:43783
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782018616648578-3598-0/krb5kdc/kudu_127.3.131.190.keytab
--principal=kudu/127.3.131.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.131.190:43783
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782018616648578-3598-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.131.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5307
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782018616648578-3598-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 21 Jun 2026 04:16:16 UTC on ccycloud.quasar-xycaha.root.comops.site
build id 4457881
I20260621 05:10:24.995487 5307 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 05:10:24.995752 5307 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260621 05:10:25.001085 5307 server_base.cc:1034] running on GCE node
W20260621 05:10:25.001003 5312 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:10:25.001003 5315 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:10:25.001032 5313 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:10:25.001647 5307 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 05:10:25.001957 5307 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 05:10:25.003151 5307 hybrid_clock.cc:648] HybridClock initialized: now 1782018625003126 us; error 43 us; skew 500 ppm
Jun 21 05:10:25 dist-test-slave-vrq1 krb5kdc[5294](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782018625, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.131.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 05:10:25.006305 5307 init.cc:377] Logged in from keytab as kudu/127.3.131.190@KRBTEST.COM (short username kudu)
I20260621 05:10:25.006872 5307 webserver.cc:460] Webserver started at http://127.3.131.190:45585/ using document root <none> and password file <none>
I20260621 05:10:25.007134 5307 fs_manager.cc:359] Metadata directory not provided
I20260621 05:10:25.007236 5307 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 05:10:25.007408 5307 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 05:10:25.008663 5307 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782018616648578-3598-0/minicluster-data/master-0/data/instance:
uuid: "b7017628c83e4962b0fb4f3348657105"
format_stamp: "Formatted at 2026-06-21 05:10:25 on dist-test-slave-vrq1"
server_key: "dc0fec11ea884e3b2da606c6cfc092b5"
server_key_iv: "bd9841ff62fd94512a13c29bf3e8c2c9"
server_key_version: "encryptionkey@0"
I20260621 05:10:25.009070 5307 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782018616648578-3598-0/minicluster-data/master-0/wal/instance:
uuid: "b7017628c83e4962b0fb4f3348657105"
format_stamp: "Formatted at 2026-06-21 05:10:25 on dist-test-slave-vrq1"
server_key: "dc0fec11ea884e3b2da606c6cfc092b5"
server_key_iv: "bd9841ff62fd94512a13c29bf3e8c2c9"
server_key_version: "encryptionkey@0"
I20260621 05:10:25.010587 5307 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260621 05:10:25.011719 5322 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 05:10:25.012009 5307 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260621 05:10:25.012172 5307 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782018616648578-3598-0/minicluster-data/master-0/data,/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782018616648578-3598-0/minicluster-data/master-0/wal
uuid: "b7017628c83e4962b0fb4f3348657105"
format_stamp: "Formatted at 2026-06-21 05:10:25 on dist-test-slave-vrq1"
server_key: "dc0fec11ea884e3b2da606c6cfc092b5"
server_key_iv: "bd9841ff62fd94512a13c29bf3e8c2c9"
server_key_version: "encryptionkey@0"
I20260621 05:10:25.012267 5307 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782018616648578-3598-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782018616648578-3598-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782018616648578-3598-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 05:10:25.022442 5307 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 05:10:25.024972 5307 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 05:10:25.025164 5307 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 05:10:25.030143 5307 rpc_server.cc:307] RPC server started. Bound to: 127.3.131.190:43783
I20260621 05:10:25.030191 5384 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.131.190:43783 every 8 connection(s)
I20260621 05:10:25.030547 5307 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782018616648578-3598-0/minicluster-data/master-0/data/info.pb
I20260621 05:10:25.031304 5385 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 05:10:25.033667 5385 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b7017628c83e4962b0fb4f3348657105: Bootstrap starting.
I20260621 05:10:25.034241 5385 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b7017628c83e4962b0fb4f3348657105: Neither blocks nor log segments found. Creating new log.
I20260621 05:10:25.034555 5385 log.cc:824] T 00000000000000000000000000000000 P b7017628c83e4962b0fb4f3348657105: Log is configured to *not* fsync() on all Append() calls
I20260621 05:10:25.035413 5385 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b7017628c83e4962b0fb4f3348657105: No bootstrap required, opened a new log
I20260621 05:10:25.035746 3598 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWZOx2_/build/release/bin/kudu as pid 5307
I20260621 05:10:25.035848 3598 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782018616648578-3598-0/minicluster-data/master-0/wal/instance
I20260621 05:10:25.036015 3598 external_mini_cluster.cc:1428] Setting key f625c63bc0a26411078c2cece5eab89f
I20260621 05:10:25.038206 5385 raft_consensus.cc:348] T 00000000000000000000000000000000 P b7017628c83e4962b0fb4f3348657105 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b7017628c83e4962b0fb4f3348657105" member_type: VOTER last_known_addr { host: "127.3.131.190" port: 43783 } }
I20260621 05:10:25.038388 5385 raft_consensus.cc:374] T 00000000000000000000000000000000 P b7017628c83e4962b0fb4f3348657105 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 05:10:25.038426 5385 raft_consensus.cc:729] T 00000000000000000000000000000000 P b7017628c83e4962b0fb4f3348657105 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b7017628c83e4962b0fb4f3348657105, State: Initialized, Role: FOLLOWER
I20260621 05:10:25.038600 5385 consensus_queue.cc:260] T 00000000000000000000000000000000 P b7017628c83e4962b0fb4f3348657105 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b7017628c83e4962b0fb4f3348657105" member_type: VOTER last_known_addr { host: "127.3.131.190" port: 43783 } }
I20260621 05:10:25.038682 5385 raft_consensus.cc:388] T 00000000000000000000000000000000 P b7017628c83e4962b0fb4f3348657105 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260621 05:10:25.038707 5385 raft_consensus.cc:482] T 00000000000000000000000000000000 P b7017628c83e4962b0fb4f3348657105 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260621 05:10:25.038810 5385 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b7017628c83e4962b0fb4f3348657105 [term 0 FOLLOWER]: Advancing to term 1
I20260621 05:10:25.039670 5385 raft_consensus.cc:504] T 00000000000000000000000000000000 P b7017628c83e4962b0fb4f3348657105 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b7017628c83e4962b0fb4f3348657105" member_type: VOTER last_known_addr { host: "127.3.131.190" port: 43783 } }
I20260621 05:10:25.039796 5385 leader_election.cc:302] T 00000000000000000000000000000000 P b7017628c83e4962b0fb4f3348657105 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b7017628c83e4962b0fb4f3348657105; no voters:
I20260621 05:10:25.039959 5385 leader_election.cc:288] T 00000000000000000000000000000000 P b7017628c83e4962b0fb4f3348657105 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260621 05:10:25.040045 5390 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b7017628c83e4962b0fb4f3348657105 [term 1 FOLLOWER]: Leader election won for term 1
I20260621 05:10:25.040199 5385 sys_catalog.cc:564] T 00000000000000000000000000000000 P b7017628c83e4962b0fb4f3348657105 [sys.catalog]: configured and running, proceeding with master startup.
I20260621 05:10:25.040274 5390 raft_consensus.cc:686] T 00000000000000000000000000000000 P b7017628c83e4962b0fb4f3348657105 [term 1 LEADER]: Becoming Leader. State: Replica: b7017628c83e4962b0fb4f3348657105, State: Running, Role: LEADER
I20260621 05:10:25.040413 5390 consensus_queue.cc:237] T 00000000000000000000000000000000 P b7017628c83e4962b0fb4f3348657105 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b7017628c83e4962b0fb4f3348657105" member_type: VOTER last_known_addr { host: "127.3.131.190" port: 43783 } }
I20260621 05:10:25.040846 5391 sys_catalog.cc:455] T 00000000000000000000000000000000 P b7017628c83e4962b0fb4f3348657105 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b7017628c83e4962b0fb4f3348657105" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b7017628c83e4962b0fb4f3348657105" member_type: VOTER last_known_addr { host: "127.3.131.190" port: 43783 } } }
I20260621 05:10:25.040833 5390 sys_catalog.cc:455] T 00000000000000000000000000000000 P b7017628c83e4962b0fb4f3348657105 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b7017628c83e4962b0fb4f3348657105. Latest consensus state: current_term: 1 leader_uuid: "b7017628c83e4962b0fb4f3348657105" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b7017628c83e4962b0fb4f3348657105" member_type: VOTER last_known_addr { host: "127.3.131.190" port: 43783 } } }
I20260621 05:10:25.041050 5391 sys_catalog.cc:458] T 00000000000000000000000000000000 P b7017628c83e4962b0fb4f3348657105 [sys.catalog]: This master's current role is: LEADER
I20260621 05:10:25.041064 5390 sys_catalog.cc:458] T 00000000000000000000000000000000 P b7017628c83e4962b0fb4f3348657105 [sys.catalog]: This master's current role is: LEADER
I20260621 05:10:25.041389 5399 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260621 05:10:25.042142 5399 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260621 05:10:25.044126 5399 catalog_manager.cc:1347] Generated new cluster ID: a9395ed02d024cf68679e111c8debf1a
I20260621 05:10:25.044189 5399 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Jun 21 05:10:25 dist-test-slave-vrq1 krb5kdc[5294](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782018624, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.131.190@KRBTEST.COM
I20260621 05:10:25.049072 5389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:10:25.036639 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39488 (local address 127.3.131.190:43783)
0621 05:10:25.036834 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0621 05:10:25.036845 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0621 05:10:25.036883 (+ 38us) server_negotiation.cc:407] Connection header received
0621 05:10:25.037814 (+ 931us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:10:25.037822 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:10:25.037881 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:10:25.038048 (+ 167us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:10:25.038890 (+ 842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:10:25.040322 (+ 1432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:10:25.041605 (+ 1283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:10:25.042252 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:10:25.042365 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:10:25.045338 (+ 2973us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:10:25.045342 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:10:25.045343 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:10:25.045356 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:10:25.047354 (+ 1998us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:10:25.047775 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:10:25.047779 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:10:25.047780 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:10:25.047899 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:10:25.048137 (+ 238us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:10:25.048138 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:10:25.048139 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:10:25.048414 (+ 275us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:10:25.048623 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0621 05:10:25.048744 (+ 121us) server_negotiation.cc:299] Negotiation successful
0621 05:10:25.048908 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":131,"thread_start_us":101,"threads_started":1}
I20260621 05:10:25.050268 5399 catalog_manager.cc:1370] Generated new certificate authority record
I20260621 05:10:25.050840 5399 catalog_manager.cc:1504] Loading token signing keys...
I20260621 05:10:25.053622 5399 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b7017628c83e4962b0fb4f3348657105: Generated new TSK 0
I20260621 05:10:25.053819 5399 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.3.131.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.131.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.131.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782018616648578-3598-0/krb5kdc/kudu_127.3.131.129.keytab.
Entry for principal kudu/127.3.131.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782018616648578-3598-0/krb5kdc/kudu_127.3.131.129.keytab.
I20260621 05:10:25.086474 3598 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWZOx2_/build/release/bin/kudu
/tmp/dist-test-taskWZOx2_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782018616648578-3598-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782018616648578-3598-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782018616648578-3598-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782018616648578-3598-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.131.129:0
--local_ip_for_outbound_sockets=127.3.131.129
--webserver_interface=127.3.131.129
--webserver_port=0
--tserver_master_addrs=127.3.131.190:43783
--keytab_file=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782018616648578-3598-0/krb5kdc/kudu_127.3.131.129.keytab
--principal=kudu/127.3.131.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.131.148:35995
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782018616648578-3598-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782018616648578-3598-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782018616648578-3598-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 05:10:25.191035 5411 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 05:10:25.191377 5411 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 05:10:25.191437 5411 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 05:10:25.192818 5411 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 05:10:25.192906 5411 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 05:10:25.192943 5411 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260621 05:10:25.192976 5411 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260621 05:10:25.193011 5411 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.131.129
I20260621 05:10:25.194701 5411 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.131.148:35995
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782018616648578-3598-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782018616648578-3598-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782018616648578-3598-0/krb5kdc/kudu_127.3.131.129.keytab
--principal=kudu/127.3.131.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.131.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782018616648578-3598-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.131.129
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.3.131.190:43783
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5411
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.131.129
--log_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782018616648578-3598-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 21 Jun 2026 04:16:16 UTC on ccycloud.quasar-xycaha.root.comops.site
build id 4457881
I20260621 05:10:25.195007 5411 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 05:10:25.195313 5411 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 05:10:25.202386 5416 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:10:25.202395 5417 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:10:25.202625 5411 server_base.cc:1034] running on GCE node
W20260621 05:10:25.202394 5419 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:10:25.202982 5411 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 05:10:25.203361 5411 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 05:10:25.204556 5411 hybrid_clock.cc:648] HybridClock initialized: now 1782018625204535 us; error 33 us; skew 500 ppm
Jun 21 05:10:25 dist-test-slave-vrq1 krb5kdc[5294](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782018625, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.131.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 05:10:25.207581 5411 init.cc:377] Logged in from keytab as kudu/127.3.131.129@KRBTEST.COM (short username kudu)
I20260621 05:10:25.208143 5411 webserver.cc:460] Webserver started at http://127.3.131.129:35291/ using document root <none> and password file <none>
I20260621 05:10:25.208436 5411 fs_manager.cc:359] Metadata directory not provided
I20260621 05:10:25.208488 5411 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 05:10:25.208662 5411 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 05:10:25.209777 5411 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782018616648578-3598-0/minicluster-data/ts-0/data/instance:
uuid: "1f4e2cd7408443c0a14c51ebb038ec73"
format_stamp: "Formatted at 2026-06-21 05:10:25 on dist-test-slave-vrq1"
server_key: "ee331a541c0b0e761f20cd4d3bc563a0"
server_key_iv: "206ab005b1641687a625ccd2d12dcb46"
server_key_version: "encryptionkey@0"
I20260621 05:10:25.210139 5411 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782018616648578-3598-0/minicluster-data/ts-0/wal/instance:
uuid: "1f4e2cd7408443c0a14c51ebb038ec73"
format_stamp: "Formatted at 2026-06-21 05:10:25 on dist-test-slave-vrq1"
server_key: "ee331a541c0b0e761f20cd4d3bc563a0"
server_key_iv: "206ab005b1641687a625ccd2d12dcb46"
server_key_version: "encryptionkey@0"
I20260621 05:10:25.211644 5411 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260621 05:10:25.212491 5426 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 05:10:25.212692 5411 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260621 05:10:25.212812 5411 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782018616648578-3598-0/minicluster-data/ts-0/data,/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782018616648578-3598-0/minicluster-data/ts-0/wal
uuid: "1f4e2cd7408443c0a14c51ebb038ec73"
format_stamp: "Formatted at 2026-06-21 05:10:25 on dist-test-slave-vrq1"
server_key: "ee331a541c0b0e761f20cd4d3bc563a0"
server_key_iv: "206ab005b1641687a625ccd2d12dcb46"
server_key_version: "encryptionkey@0"
I20260621 05:10:25.212880 5411 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782018616648578-3598-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782018616648578-3598-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782018616648578-3598-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 05:10:25.226284 5411 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 05:10:25.228974 5411 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 05:10:25.229164 5411 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 05:10:25.229708 5411 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 05:10:25.229746 5411 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 05:10:25.229770 5411 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 05:10:25.229787 5411 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 05:10:25.237564 5411 rpc_server.cc:307] RPC server started. Bound to: 127.3.131.129:45955
I20260621 05:10:25.237624 5541 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.131.129:45955 every 8 connection(s)
I20260621 05:10:25.237962 5411 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782018616648578-3598-0/minicluster-data/ts-0/data/info.pb
Jun 21 05:10:25 dist-test-slave-vrq1 krb5kdc[5294](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782018625, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.131.129@KRBTEST.COM for kudu/127.3.131.190@KRBTEST.COM
I20260621 05:10:25.241963 5389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:10:25.230346 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.131.129:39681 (local address 127.3.131.190:43783)
0621 05:10:25.230391 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0621 05:10:25.230395 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 05:10:25.230681 (+ 286us) server_negotiation.cc:407] Connection header received
0621 05:10:25.231329 (+ 648us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:10:25.231330 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:10:25.231343 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:10:25.231423 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:10:25.232480 (+ 1057us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:10:25.233378 (+ 898us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:10:25.235193 (+ 1815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:10:25.235520 (+ 327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:10:25.235582 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:10:25.238725 (+ 3143us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:10:25.238726 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:10:25.238727 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:10:25.238734 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:10:25.240244 (+ 1510us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:10:25.240796 (+ 552us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:10:25.240798 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:10:25.240798 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:10:25.240870 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:10:25.241166 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:10:25.241167 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:10:25.241167 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:10:25.241356 (+ 189us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:10:25.241446 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0621 05:10:25.241761 (+ 315us) server_negotiation.cc:299] Negotiation successful
0621 05:10:25.241867 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260621 05:10:25.242164 5435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:10:25.230359 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.131.190:43783 (local address 127.3.131.129:39681)
0621 05:10:25.230527 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0621 05:10:25.230565 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0621 05:10:25.231207 (+ 642us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:10:25.231561 (+ 354us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:10:25.231567 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:10:25.232018 (+ 451us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:10:25.232355 (+ 337us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:10:25.232362 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:10:25.233510 (+ 1148us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:10:25.233512 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:10:25.235054 (+ 1542us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:10:25.235058 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:10:25.235619 (+ 561us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:10:25.235620 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:10:25.235873 (+ 253us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:10:25.236603 (+ 730us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 05:10:25.236617 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:10:25.238581 (+ 1964us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 05:10:25.240341 (+ 1760us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:10:25.240349 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:10:25.240352 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:10:25.240708 (+ 356us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:10:25.240947 (+ 239us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:10:25.240948 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:10:25.240948 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:10:25.241088 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:10:25.241448 (+ 360us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 05:10:25.241456 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 05:10:25.241620 (+ 164us) client_negotiation.cc:769] Sending connection context
0621 05:10:25.241738 (+ 118us) client_negotiation.cc:240] Negotiation successful
0621 05:10:25.241913 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":100,"threads_started":1}
I20260621 05:10:25.242789 5542 heartbeater.cc:347] Connected to a master server at 127.3.131.190:43783
I20260621 05:10:25.242880 5542 heartbeater.cc:464] Registering TS with master...
I20260621 05:10:25.242944 3598 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWZOx2_/build/release/bin/kudu as pid 5411
I20260621 05:10:25.243044 3598 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782018616648578-3598-0/minicluster-data/ts-0/wal/instance
I20260621 05:10:25.243140 5542 heartbeater.cc:511] Master 127.3.131.190:43783 requested a full tablet report, sending...
I20260621 05:10:25.243278 3598 external_mini_cluster.cc:1428] Setting key c419307e3621245c350ae76711ef498a
I20260621 05:10:25.243793 5338 ts_manager.cc:194] Registered new tserver with Master: 1f4e2cd7408443c0a14c51ebb038ec73 (127.3.131.129:45955)
I20260621 05:10:25.244617 5338 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.131.129@KRBTEST.COM'} at 127.3.131.129:39681
I20260621 05:10:25.251492 5548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:10:25.244069 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.131.190:43783 (local address 127.3.131.129:41391)
0621 05:10:25.244229 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0621 05:10:25.244241 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0621 05:10:25.244292 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:10:25.245016 (+ 724us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:10:25.245017 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:10:25.245217 (+ 200us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:10:25.245281 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:10:25.245283 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:10:25.246284 (+ 1001us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:10:25.246285 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:10:25.246802 (+ 517us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:10:25.246804 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:10:25.247337 (+ 533us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:10:25.247337 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:10:25.247400 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:10:25.247957 (+ 557us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 05:10:25.247965 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:10:25.248329 (+ 364us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 05:10:25.250343 (+ 2014us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:10:25.250344 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:10:25.250345 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:10:25.250530 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:10:25.250754 (+ 224us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:10:25.250755 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:10:25.250755 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:10:25.250812 (+ 57us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:10:25.251124 (+ 312us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 05:10:25.251125 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 05:10:25.251170 (+ 45us) client_negotiation.cc:769] Sending connection context
0621 05:10:25.251306 (+ 136us) client_negotiation.cc:240] Negotiation successful
0621 05:10:25.251404 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":80,"threads_started":1}
I20260621 05:10:25.251544 5389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:10:25.244829 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.131.129:41391 (local address 127.3.131.190:43783)
0621 05:10:25.244886 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0621 05:10:25.244888 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:10:25.244898 (+ 10us) server_negotiation.cc:407] Connection header received
0621 05:10:25.244921 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:10:25.244922 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:10:25.244933 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:10:25.244995 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:10:25.245376 (+ 381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:10:25.246165 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:10:25.246891 (+ 726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:10:25.247259 (+ 368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:10:25.247330 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:10:25.248421 (+ 1091us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:10:25.248422 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:10:25.248423 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:10:25.248430 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:10:25.250247 (+ 1817us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:10:25.250610 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:10:25.250612 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:10:25.250612 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:10:25.250681 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:10:25.250877 (+ 196us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:10:25.250878 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:10:25.250878 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:10:25.251034 (+ 156us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:10:25.251117 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0621 05:10:25.251350 (+ 233us) server_negotiation.cc:299] Negotiation successful
0621 05:10:25.251446 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
WARNING: no policy specified for kudu/127.3.131.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.131.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.131.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782018616648578-3598-0/krb5kdc/kudu_127.3.131.130.keytab.
Entry for principal kudu/127.3.131.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782018616648578-3598-0/krb5kdc/kudu_127.3.131.130.keytab.
I20260621 05:10:25.271111 3598 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWZOx2_/build/release/bin/kudu
/tmp/dist-test-taskWZOx2_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782018616648578-3598-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782018616648578-3598-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782018616648578-3598-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782018616648578-3598-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.131.130:0
--local_ip_for_outbound_sockets=127.3.131.130
--webserver_interface=127.3.131.130
--webserver_port=0
--tserver_master_addrs=127.3.131.190:43783
--keytab_file=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782018616648578-3598-0/krb5kdc/kudu_127.3.131.130.keytab
--principal=kudu/127.3.131.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.131.148:35995
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782018616648578-3598-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782018616648578-3598-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782018616648578-3598-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 05:10:25.359848 5551 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 05:10:25.360087 5551 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 05:10:25.360117 5551 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 05:10:25.361361 5551 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 05:10:25.361404 5551 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 05:10:25.361426 5551 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260621 05:10:25.361439 5551 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260621 05:10:25.361456 5551 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.131.130
I20260621 05:10:25.363077 5551 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.131.148:35995
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782018616648578-3598-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782018616648578-3598-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782018616648578-3598-0/krb5kdc/kudu_127.3.131.130.keytab
--principal=kudu/127.3.131.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.131.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782018616648578-3598-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.131.130
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.3.131.190:43783
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5551
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.131.130
--log_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782018616648578-3598-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 21 Jun 2026 04:16:16 UTC on ccycloud.quasar-xycaha.root.comops.site
build id 4457881
I20260621 05:10:25.363372 5551 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 05:10:25.363587 5551 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 05:10:25.369105 5556 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:10:25.369102 5559 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:10:25.369102 5557 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:10:25.369433 5551 server_base.cc:1034] running on GCE node
I20260621 05:10:25.369663 5551 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 05:10:25.369916 5551 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 05:10:25.371124 5551 hybrid_clock.cc:648] HybridClock initialized: now 1782018625371072 us; error 32 us; skew 500 ppm
Jun 21 05:10:25 dist-test-slave-vrq1 krb5kdc[5294](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782018625, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.131.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 05:10:25.374043 5551 init.cc:377] Logged in from keytab as kudu/127.3.131.130@KRBTEST.COM (short username kudu)
I20260621 05:10:25.374589 5551 webserver.cc:460] Webserver started at http://127.3.131.130:40537/ using document root <none> and password file <none>
I20260621 05:10:25.374851 5551 fs_manager.cc:359] Metadata directory not provided
I20260621 05:10:25.374925 5551 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 05:10:25.375080 5551 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 05:10:25.376497 5551 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782018616648578-3598-0/minicluster-data/ts-1/data/instance:
uuid: "71d95c905a8f4ff181c1de9655d4a593"
format_stamp: "Formatted at 2026-06-21 05:10:25 on dist-test-slave-vrq1"
server_key: "f9824fb652198bde4fad2fbe854bd13b"
server_key_iv: "82c9ccf0690a2f9193b6524ef2ea1a96"
server_key_version: "encryptionkey@0"
I20260621 05:10:25.376878 5551 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782018616648578-3598-0/minicluster-data/ts-1/wal/instance:
uuid: "71d95c905a8f4ff181c1de9655d4a593"
format_stamp: "Formatted at 2026-06-21 05:10:25 on dist-test-slave-vrq1"
server_key: "f9824fb652198bde4fad2fbe854bd13b"
server_key_iv: "82c9ccf0690a2f9193b6524ef2ea1a96"
server_key_version: "encryptionkey@0"
I20260621 05:10:25.378353 5551 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260621 05:10:25.379271 5566 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 05:10:25.379464 5551 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260621 05:10:25.379568 5551 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782018616648578-3598-0/minicluster-data/ts-1/data,/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782018616648578-3598-0/minicluster-data/ts-1/wal
uuid: "71d95c905a8f4ff181c1de9655d4a593"
format_stamp: "Formatted at 2026-06-21 05:10:25 on dist-test-slave-vrq1"
server_key: "f9824fb652198bde4fad2fbe854bd13b"
server_key_iv: "82c9ccf0690a2f9193b6524ef2ea1a96"
server_key_version: "encryptionkey@0"
I20260621 05:10:25.379657 5551 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782018616648578-3598-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782018616648578-3598-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782018616648578-3598-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 05:10:25.397722 5551 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 05:10:25.400069 5551 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 05:10:25.400271 5551 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 05:10:25.400864 5551 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 05:10:25.400921 5551 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 05:10:25.400961 5551 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 05:10:25.400979 5551 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 05:10:25.407701 5551 rpc_server.cc:307] RPC server started. Bound to: 127.3.131.130:35573
I20260621 05:10:25.407783 5681 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.131.130:35573 every 8 connection(s)
I20260621 05:10:25.408118 5551 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782018616648578-3598-0/minicluster-data/ts-1/data/info.pb
Jun 21 05:10:25 dist-test-slave-vrq1 krb5kdc[5294](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782018625, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.131.130@KRBTEST.COM for kudu/127.3.131.190@KRBTEST.COM
I20260621 05:10:25.412209 5389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:10:25.401453 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.131.130:36501 (local address 127.3.131.190:43783)
0621 05:10:25.401536 (+ 83us) server_negotiation.cc:206] Beginning negotiation
0621 05:10:25.401539 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0621 05:10:25.401748 (+ 209us) server_negotiation.cc:407] Connection header received
0621 05:10:25.402326 (+ 578us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:10:25.402328 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:10:25.402340 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:10:25.402401 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:10:25.403367 (+ 966us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:10:25.404239 (+ 872us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:10:25.405845 (+ 1606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:10:25.406200 (+ 355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:10:25.406292 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:10:25.409200 (+ 2908us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:10:25.409202 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:10:25.409202 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:10:25.409209 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:10:25.410727 (+ 1518us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:10:25.411306 (+ 579us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:10:25.411307 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:10:25.411307 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:10:25.411363 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:10:25.411614 (+ 251us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:10:25.411614 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:10:25.411614 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:10:25.411732 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:10:25.411782 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0621 05:10:25.412067 (+ 285us) server_negotiation.cc:299] Negotiation successful
0621 05:10:25.412136 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260621 05:10:25.412563 5575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:10:25.401424 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.131.190:43783 (local address 127.3.131.130:36501)
0621 05:10:25.401616 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0621 05:10:25.401658 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0621 05:10:25.402225 (+ 567us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:10:25.402479 (+ 254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:10:25.402484 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:10:25.402921 (+ 437us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:10:25.403258 (+ 337us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:10:25.403266 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:10:25.404351 (+ 1085us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:10:25.404352 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:10:25.405748 (+ 1396us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:10:25.405751 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:10:25.406300 (+ 549us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:10:25.406301 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:10:25.406459 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:10:25.407057 (+ 598us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 05:10:25.407074 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:10:25.408994 (+ 1920us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 05:10:25.410834 (+ 1840us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:10:25.410842 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:10:25.410844 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:10:25.411117 (+ 273us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:10:25.411428 (+ 311us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:10:25.411429 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:10:25.411429 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:10:25.411550 (+ 121us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:10:25.411790 (+ 240us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 05:10:25.411795 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 05:10:25.411928 (+ 133us) client_negotiation.cc:769] Sending connection context
0621 05:10:25.412016 (+ 88us) client_negotiation.cc:240] Negotiation successful
0621 05:10:25.412154 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":151,"thread_start_us":115,"threads_started":1}
I20260621 05:10:25.413240 5682 heartbeater.cc:347] Connected to a master server at 127.3.131.190:43783
I20260621 05:10:25.413337 5682 heartbeater.cc:464] Registering TS with master...
I20260621 05:10:25.413532 5682 heartbeater.cc:511] Master 127.3.131.190:43783 requested a full tablet report, sending...
I20260621 05:10:25.413942 5338 ts_manager.cc:194] Registered new tserver with Master: 71d95c905a8f4ff181c1de9655d4a593 (127.3.131.130:35573)
I20260621 05:10:25.414343 5338 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.131.130@KRBTEST.COM'} at 127.3.131.130:36501
I20260621 05:10:25.417272 3598 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWZOx2_/build/release/bin/kudu as pid 5551
I20260621 05:10:25.417382 3598 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782018616648578-3598-0/minicluster-data/ts-1/wal/instance
I20260621 05:10:25.417591 3598 external_mini_cluster.cc:1428] Setting key d3a8659c7833a1f465870594af61fb11
I20260621 05:10:25.421106 5389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:10:25.414538 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.131.130:41017 (local address 127.3.131.190:43783)
0621 05:10:25.414580 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0621 05:10:25.414582 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:10:25.414783 (+ 201us) server_negotiation.cc:407] Connection header received
0621 05:10:25.414825 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:10:25.414826 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:10:25.414836 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:10:25.414885 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:10:25.415292 (+ 407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:10:25.415992 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:10:25.416802 (+ 810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:10:25.417045 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:10:25.417104 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:10:25.418313 (+ 1209us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:10:25.418313 (+ 0us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:10:25.418314 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:10:25.418319 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:10:25.419728 (+ 1409us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:10:25.420171 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:10:25.420173 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:10:25.420173 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:10:25.420230 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:10:25.420461 (+ 231us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:10:25.420462 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:10:25.420462 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:10:25.420652 (+ 190us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:10:25.420721 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0621 05:10:25.420959 (+ 238us) server_negotiation.cc:299] Negotiation successful
0621 05:10:25.421041 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260621 05:10:25.421281 5688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:10:25.414504 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.131.190:43783 (local address 127.3.131.130:41017)
0621 05:10:25.414699 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0621 05:10:25.414710 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0621 05:10:25.414773 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:10:25.414891 (+ 118us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:10:25.414891 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:10:25.415109 (+ 218us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:10:25.415208 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:10:25.415210 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:10:25.416089 (+ 879us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:10:25.416090 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:10:25.416725 (+ 635us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:10:25.416728 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:10:25.417125 (+ 397us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:10:25.417126 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:10:25.417216 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:10:25.417816 (+ 600us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 05:10:25.417821 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:10:25.418218 (+ 397us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 05:10:25.419831 (+ 1613us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:10:25.419832 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:10:25.419833 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:10:25.420095 (+ 262us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:10:25.420304 (+ 209us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:10:25.420305 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:10:25.420305 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:10:25.420375 (+ 70us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:10:25.420731 (+ 356us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 05:10:25.420732 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 05:10:25.420807 (+ 75us) client_negotiation.cc:769] Sending connection context
0621 05:10:25.420896 (+ 89us) client_negotiation.cc:240] Negotiation successful
0621 05:10:25.421015 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":167,"thread_start_us":111,"threads_started":1}
WARNING: no policy specified for kudu/127.3.131.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.131.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.131.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782018616648578-3598-0/krb5kdc/kudu_127.3.131.131.keytab.
Entry for principal kudu/127.3.131.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782018616648578-3598-0/krb5kdc/kudu_127.3.131.131.keytab.
I20260621 05:10:25.447861 3598 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWZOx2_/build/release/bin/kudu
/tmp/dist-test-taskWZOx2_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782018616648578-3598-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782018616648578-3598-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782018616648578-3598-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782018616648578-3598-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.131.131:0
--local_ip_for_outbound_sockets=127.3.131.131
--webserver_interface=127.3.131.131
--webserver_port=0
--tserver_master_addrs=127.3.131.190:43783
--keytab_file=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782018616648578-3598-0/krb5kdc/kudu_127.3.131.131.keytab
--principal=kudu/127.3.131.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.131.148:35995
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782018616648578-3598-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782018616648578-3598-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782018616648578-3598-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 05:10:25.535228 5691 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 05:10:25.535516 5691 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 05:10:25.535557 5691 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 05:10:25.536738 5691 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 05:10:25.536779 5691 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 05:10:25.536799 5691 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260621 05:10:25.536813 5691 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260621 05:10:25.536829 5691 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.131.131
I20260621 05:10:25.538198 5691 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.131.148:35995
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782018616648578-3598-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782018616648578-3598-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782018616648578-3598-0/krb5kdc/kudu_127.3.131.131.keytab
--principal=kudu/127.3.131.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.131.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782018616648578-3598-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.131.131
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.3.131.190:43783
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5691
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.131.131
--log_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782018616648578-3598-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 21 Jun 2026 04:16:16 UTC on ccycloud.quasar-xycaha.root.comops.site
build id 4457881
I20260621 05:10:25.538383 5691 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 05:10:25.538558 5691 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 05:10:25.544029 5697 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:10:25.544029 5696 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:10:25.544219 5691 server_base.cc:1034] running on GCE node
W20260621 05:10:25.544054 5699 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:10:25.544510 5691 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 05:10:25.544806 5691 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 05:10:25.545980 5691 hybrid_clock.cc:648] HybridClock initialized: now 1782018625545953 us; error 44 us; skew 500 ppm
Jun 21 05:10:25 dist-test-slave-vrq1 krb5kdc[5294](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782018625, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.131.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 05:10:25.548784 5691 init.cc:377] Logged in from keytab as kudu/127.3.131.131@KRBTEST.COM (short username kudu)
I20260621 05:10:25.549368 5691 webserver.cc:460] Webserver started at http://127.3.131.131:38975/ using document root <none> and password file <none>
I20260621 05:10:25.549626 5691 fs_manager.cc:359] Metadata directory not provided
I20260621 05:10:25.549734 5691 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 05:10:25.549880 5691 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 05:10:25.550940 5691 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782018616648578-3598-0/minicluster-data/ts-2/data/instance:
uuid: "8571114a663448e99821d16f67d38bcb"
format_stamp: "Formatted at 2026-06-21 05:10:25 on dist-test-slave-vrq1"
server_key: "909b5f6c31ea6a75c63cb6f9a9436332"
server_key_iv: "315fb808483905bbcdaf95d0567a40ba"
server_key_version: "encryptionkey@0"
I20260621 05:10:25.551362 5691 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782018616648578-3598-0/minicluster-data/ts-2/wal/instance:
uuid: "8571114a663448e99821d16f67d38bcb"
format_stamp: "Formatted at 2026-06-21 05:10:25 on dist-test-slave-vrq1"
server_key: "909b5f6c31ea6a75c63cb6f9a9436332"
server_key_iv: "315fb808483905bbcdaf95d0567a40ba"
server_key_version: "encryptionkey@0"
I20260621 05:10:25.552788 5691 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260621 05:10:25.553737 5706 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 05:10:25.553956 5691 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260621 05:10:25.554057 5691 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782018616648578-3598-0/minicluster-data/ts-2/data,/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782018616648578-3598-0/minicluster-data/ts-2/wal
uuid: "8571114a663448e99821d16f67d38bcb"
format_stamp: "Formatted at 2026-06-21 05:10:25 on dist-test-slave-vrq1"
server_key: "909b5f6c31ea6a75c63cb6f9a9436332"
server_key_iv: "315fb808483905bbcdaf95d0567a40ba"
server_key_version: "encryptionkey@0"
I20260621 05:10:25.554148 5691 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782018616648578-3598-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782018616648578-3598-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782018616648578-3598-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 05:10:25.569278 5691 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 05:10:25.572059 5691 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 05:10:25.572280 5691 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 05:10:25.572893 5691 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 05:10:25.572969 5691 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 05:10:25.573024 5691 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 05:10:25.573051 5691 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 21 05:10:25 dist-test-slave-vrq1 krb5kdc[5294](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782018625, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.131.131@KRBTEST.COM for kudu/127.3.131.190@KRBTEST.COM
I20260621 05:10:25.581326 5691 rpc_server.cc:307] RPC server started. Bound to: 127.3.131.131:43851
I20260621 05:10:25.581411 5821 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.131.131:43851 every 8 connection(s)
I20260621 05:10:25.581794 5691 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782018616648578-3598-0/minicluster-data/ts-2/data/info.pb
I20260621 05:10:25.583527 3598 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWZOx2_/build/release/bin/kudu as pid 5691
I20260621 05:10:25.583653 3598 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782018616648578-3598-0/minicluster-data/ts-2/wal/instance
I20260621 05:10:25.583925 3598 external_mini_cluster.cc:1428] Setting key bab175461bc0405fec169cd383694918
I20260621 05:10:25.584822 5389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:10:25.573488 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.131.131:55803 (local address 127.3.131.190:43783)
0621 05:10:25.573570 (+ 82us) server_negotiation.cc:206] Beginning negotiation
0621 05:10:25.573573 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0621 05:10:25.573823 (+ 250us) server_negotiation.cc:407] Connection header received
0621 05:10:25.574345 (+ 522us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:10:25.574347 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:10:25.574361 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:10:25.574457 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:10:25.575485 (+ 1028us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:10:25.576442 (+ 957us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:10:25.577598 (+ 1156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:10:25.577802 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:10:25.577882 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:10:25.581031 (+ 3149us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:10:25.581033 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:10:25.581033 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:10:25.581044 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:10:25.582844 (+ 1800us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:10:25.583425 (+ 581us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:10:25.583427 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:10:25.583427 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:10:25.583532 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:10:25.583884 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:10:25.583885 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:10:25.583886 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:10:25.584063 (+ 177us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:10:25.584207 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0621 05:10:25.584629 (+ 422us) server_negotiation.cc:299] Negotiation successful
0621 05:10:25.584725 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260621 05:10:25.585064 5715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:10:25.573463 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.131.190:43783 (local address 127.3.131.131:55803)
0621 05:10:25.573677 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0621 05:10:25.573722 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0621 05:10:25.574241 (+ 519us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:10:25.574623 (+ 382us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:10:25.574628 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:10:25.575011 (+ 383us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:10:25.575353 (+ 342us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:10:25.575359 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:10:25.576558 (+ 1199us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:10:25.576559 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:10:25.577492 (+ 933us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:10:25.577494 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:10:25.577903 (+ 409us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:10:25.577904 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:10:25.578048 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:10:25.578725 (+ 677us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 05:10:25.578739 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:10:25.580909 (+ 2170us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 05:10:25.582930 (+ 2021us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:10:25.582936 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:10:25.582937 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:10:25.583348 (+ 411us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:10:25.583631 (+ 283us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:10:25.583632 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:10:25.583633 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:10:25.583789 (+ 156us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:10:25.584189 (+ 400us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 05:10:25.584197 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 05:10:25.584451 (+ 254us) client_negotiation.cc:769] Sending connection context
0621 05:10:25.584574 (+ 123us) client_negotiation.cc:240] Negotiation successful
0621 05:10:25.584771 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":169,"thread_start_us":110,"threads_started":1}
I20260621 05:10:25.585613 5822 heartbeater.cc:347] Connected to a master server at 127.3.131.190:43783
I20260621 05:10:25.585712 5822 heartbeater.cc:464] Registering TS with master...
I20260621 05:10:25.585927 5822 heartbeater.cc:511] Master 127.3.131.190:43783 requested a full tablet report, sending...
I20260621 05:10:25.586344 5339 ts_manager.cc:194] Registered new tserver with Master: 8571114a663448e99821d16f67d38bcb (127.3.131.131:43851)
I20260621 05:10:25.586819 5339 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.131.131@KRBTEST.COM'} at 127.3.131.131:55803
I20260621 05:10:25.593236 5389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:10:25.586839 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.131.131:33549 (local address 127.3.131.190:43783)
0621 05:10:25.586881 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0621 05:10:25.586883 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:10:25.586996 (+ 113us) server_negotiation.cc:407] Connection header received
0621 05:10:25.587078 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:10:25.587078 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:10:25.587088 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:10:25.587133 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:10:25.587528 (+ 395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:10:25.588040 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:10:25.588793 (+ 753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:10:25.588989 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:10:25.589074 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:10:25.590274 (+ 1200us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:10:25.590276 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:10:25.590276 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:10:25.590290 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:10:25.591921 (+ 1631us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:10:25.592387 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:10:25.592388 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:10:25.592389 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:10:25.592454 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:10:25.592666 (+ 212us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:10:25.592667 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:10:25.592667 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:10:25.592789 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:10:25.592866 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0621 05:10:25.593041 (+ 175us) server_negotiation.cc:299] Negotiation successful
0621 05:10:25.593134 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260621 05:10:25.593264 5828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:10:25.586742 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.131.190:43783 (local address 127.3.131.131:33549)
0621 05:10:25.586934 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0621 05:10:25.586946 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0621 05:10:25.587022 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:10:25.587137 (+ 115us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:10:25.587137 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:10:25.587363 (+ 226us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:10:25.587433 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:10:25.587435 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:10:25.588142 (+ 707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:10:25.588143 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:10:25.588693 (+ 550us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:10:25.588695 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:10:25.589077 (+ 382us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:10:25.589077 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:10:25.589167 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:10:25.589729 (+ 562us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 05:10:25.589734 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:10:25.590141 (+ 407us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 05:10:25.592046 (+ 1905us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:10:25.592048 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:10:25.592048 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:10:25.592302 (+ 254us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:10:25.592525 (+ 223us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:10:25.592526 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:10:25.592527 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:10:25.592590 (+ 63us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:10:25.592856 (+ 266us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 05:10:25.592857 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 05:10:25.592924 (+ 67us) client_negotiation.cc:769] Sending connection context
0621 05:10:25.593011 (+ 87us) client_negotiation.cc:240] Negotiation successful
0621 05:10:25.593136 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":160,"thread_start_us":90,"threads_started":1}
I20260621 05:10:25.596010 3598 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 21 05:10:25 dist-test-slave-vrq1 krb5kdc[5294](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782018625, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 21 05:10:25 dist-test-slave-vrq1 krb5kdc[5294](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782018625, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.131.190@KRBTEST.COM
I20260621 05:10:25.621629 5389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:10:25.613323 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39492 (local address 127.3.131.190:43783)
0621 05:10:25.613400 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0621 05:10:25.613402 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:10:25.613409 (+ 7us) server_negotiation.cc:407] Connection header received
0621 05:10:25.613432 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:10:25.613433 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:10:25.613442 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:10:25.613484 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:10:25.614174 (+ 690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:10:25.614784 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:10:25.615786 (+ 1002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:10:25.615978 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:10:25.616123 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:10:25.618728 (+ 2605us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:10:25.618729 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:10:25.618730 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:10:25.618737 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:10:25.620226 (+ 1489us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:10:25.620593 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:10:25.620595 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:10:25.620595 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:10:25.620654 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:10:25.620907 (+ 253us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:10:25.620908 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:10:25.620908 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:10:25.621033 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:10:25.621165 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0621 05:10:25.621390 (+ 225us) server_negotiation.cc:299] Negotiation successful
0621 05:10:25.621466 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260621 05:10:25.623591 5339 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39492:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260621 05:10:25.625289 5339 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260621 05:10:25.632423 5842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:10:25.628034 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48470 (local address 127.3.131.130:35573)
0621 05:10:25.628774 (+ 740us) server_negotiation.cc:206] Beginning negotiation
0621 05:10:25.628779 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 05:10:25.628787 (+ 8us) server_negotiation.cc:407] Connection header received
0621 05:10:25.628823 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:10:25.628825 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:10:25.628841 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:10:25.628914 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 05:10:25.629069 (+ 155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:10:25.629782 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:10:25.631457 (+ 1675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:10:25.631977 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:10:25.632087 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:10:25.632150 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0621 05:10:25.632286 (+ 136us) server_negotiation.cc:299] Negotiation successful
0621 05:10:25.632325 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":703,"thread_start_us":89,"threads_started":1}
I20260621 05:10:25.632411 5840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:10:25.627924 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.131.130:35573 (local address 127.0.0.1:48470)
0621 05:10:25.628238 (+ 314us) negotiation.cc:107] Waiting for socket to connect
0621 05:10:25.628270 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0621 05:10:25.628339 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:10:25.628929 (+ 590us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:10:25.628929 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:10:25.628931 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 05:10:25.628972 (+ 41us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:10:25.628973 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:10:25.629871 (+ 898us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:10:25.629872 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:10:25.631357 (+ 1485us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:10:25.631358 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:10:25.632090 (+ 732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:10:25.632091 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:10:25.632170 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:10:25.632183 (+ 13us) client_negotiation.cc:769] Sending connection context
0621 05:10:25.632255 (+ 72us) client_negotiation.cc:240] Negotiation successful
0621 05:10:25.632312 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":289,"mutex_wait_us":90,"thread_start_us":95,"threads_started":1}
I20260621 05:10:25.632658 5843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:10:25.628114 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.131.129:45955 (local address 127.0.0.1:47102)
0621 05:10:25.628238 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0621 05:10:25.628270 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0621 05:10:25.628339 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:10:25.628974 (+ 635us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:10:25.628974 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:10:25.628976 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 05:10:25.629006 (+ 30us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:10:25.629008 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:10:25.630243 (+ 1235us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:10:25.630244 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:10:25.631334 (+ 1090us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:10:25.631337 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:10:25.632366 (+ 1029us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:10:25.632368 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:10:25.632463 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:10:25.632466 (+ 3us) client_negotiation.cc:769] Sending connection context
0621 05:10:25.632513 (+ 47us) client_negotiation.cc:240] Negotiation successful
0621 05:10:25.632570 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":104,"mutex_wait_us":2,"thread_start_us":78,"threads_started":1}
I20260621 05:10:25.633047 5844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:10:25.628337 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47102 (local address 127.3.131.129:45955)
0621 05:10:25.628513 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0621 05:10:25.628518 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 05:10:25.628526 (+ 8us) server_negotiation.cc:407] Connection header received
0621 05:10:25.628565 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:10:25.628567 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:10:25.628582 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:10:25.628657 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 05:10:25.629091 (+ 434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:10:25.630157 (+ 1066us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:10:25.631421 (+ 1264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:10:25.632222 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:10:25.632279 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:10:25.632335 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0621 05:10:25.632869 (+ 534us) server_negotiation.cc:299] Negotiation successful
0621 05:10:25.632916 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":149,"thread_start_us":79,"threads_started":1}
I20260621 05:10:25.633229 5616 tablet_service.cc:1467] Processing CreateTablet for tablet a49392eeeab6454eb7baee169e7f3a28 (DEFAULT_TABLE table=test-table [id=e7d49a9b4b9c4111a192f51649fb628f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 05:10:25.633477 5841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:10:25.628002 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51708 (local address 127.3.131.131:43851)
0621 05:10:25.628153 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0621 05:10:25.628158 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 05:10:25.628337 (+ 179us) server_negotiation.cc:407] Connection header received
0621 05:10:25.628544 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:10:25.628547 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:10:25.628565 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:10:25.628638 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 05:10:25.628855 (+ 217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:10:25.629980 (+ 1125us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:10:25.631471 (+ 1491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:10:25.632191 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:10:25.632279 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:10:25.632335 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0621 05:10:25.633270 (+ 935us) server_negotiation.cc:299] Negotiation successful
0621 05:10:25.633333 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":113,"thread_start_us":134,"threads_started":1}
I20260621 05:10:25.633574 5616 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a49392eeeab6454eb7baee169e7f3a28. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 05:10:25.633082 5839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:10:25.627883 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.131.131:43851 (local address 127.0.0.1:51708)
0621 05:10:25.628223 (+ 340us) negotiation.cc:107] Waiting for socket to connect
0621 05:10:25.628268 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0621 05:10:25.628371 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:10:25.628658 (+ 287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:10:25.628660 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:10:25.628663 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 05:10:25.628795 (+ 132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:10:25.628798 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:10:25.630062 (+ 1264us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:10:25.630062 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:10:25.631355 (+ 1293us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:10:25.631358 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:10:25.632369 (+ 1011us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:10:25.632370 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:10:25.632459 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:10:25.632463 (+ 4us) client_negotiation.cc:769] Sending connection context
0621 05:10:25.632964 (+ 501us) client_negotiation.cc:240] Negotiation successful
0621 05:10:25.633014 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":312,"mutex_wait_us":68,"thread_start_us":79,"threads_started":1}
I20260621 05:10:25.633757 5476 tablet_service.cc:1467] Processing CreateTablet for tablet a49392eeeab6454eb7baee169e7f3a28 (DEFAULT_TABLE table=test-table [id=e7d49a9b4b9c4111a192f51649fb628f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 05:10:25.634127 5476 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a49392eeeab6454eb7baee169e7f3a28. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 05:10:25.634526 5756 tablet_service.cc:1467] Processing CreateTablet for tablet a49392eeeab6454eb7baee169e7f3a28 (DEFAULT_TABLE table=test-table [id=e7d49a9b4b9c4111a192f51649fb628f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 05:10:25.634853 5756 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a49392eeeab6454eb7baee169e7f3a28. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 05:10:25.637262 5845 tablet_bootstrap.cc:492] T a49392eeeab6454eb7baee169e7f3a28 P 1f4e2cd7408443c0a14c51ebb038ec73: Bootstrap starting.
I20260621 05:10:25.637710 5846 tablet_bootstrap.cc:492] T a49392eeeab6454eb7baee169e7f3a28 P 71d95c905a8f4ff181c1de9655d4a593: Bootstrap starting.
I20260621 05:10:25.637946 5845 tablet_bootstrap.cc:654] T a49392eeeab6454eb7baee169e7f3a28 P 1f4e2cd7408443c0a14c51ebb038ec73: Neither blocks nor log segments found. Creating new log.
I20260621 05:10:25.638329 5846 tablet_bootstrap.cc:654] T a49392eeeab6454eb7baee169e7f3a28 P 71d95c905a8f4ff181c1de9655d4a593: Neither blocks nor log segments found. Creating new log.
I20260621 05:10:25.638319 5845 log.cc:824] T a49392eeeab6454eb7baee169e7f3a28 P 1f4e2cd7408443c0a14c51ebb038ec73: Log is configured to *not* fsync() on all Append() calls
I20260621 05:10:25.638322 5847 tablet_bootstrap.cc:492] T a49392eeeab6454eb7baee169e7f3a28 P 8571114a663448e99821d16f67d38bcb: Bootstrap starting.
I20260621 05:10:25.638674 5846 log.cc:824] T a49392eeeab6454eb7baee169e7f3a28 P 71d95c905a8f4ff181c1de9655d4a593: Log is configured to *not* fsync() on all Append() calls
I20260621 05:10:25.639415 5847 tablet_bootstrap.cc:654] T a49392eeeab6454eb7baee169e7f3a28 P 8571114a663448e99821d16f67d38bcb: Neither blocks nor log segments found. Creating new log.
I20260621 05:10:25.639561 5846 tablet_bootstrap.cc:492] T a49392eeeab6454eb7baee169e7f3a28 P 71d95c905a8f4ff181c1de9655d4a593: No bootstrap required, opened a new log
I20260621 05:10:25.639635 5846 ts_tablet_manager.cc:1397] T a49392eeeab6454eb7baee169e7f3a28 P 71d95c905a8f4ff181c1de9655d4a593: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260621 05:10:25.639748 5845 tablet_bootstrap.cc:492] T a49392eeeab6454eb7baee169e7f3a28 P 1f4e2cd7408443c0a14c51ebb038ec73: No bootstrap required, opened a new log
I20260621 05:10:25.639810 5845 ts_tablet_manager.cc:1397] T a49392eeeab6454eb7baee169e7f3a28 P 1f4e2cd7408443c0a14c51ebb038ec73: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260621 05:10:25.639797 5847 log.cc:824] T a49392eeeab6454eb7baee169e7f3a28 P 8571114a663448e99821d16f67d38bcb: Log is configured to *not* fsync() on all Append() calls
I20260621 05:10:25.640547 5847 tablet_bootstrap.cc:492] T a49392eeeab6454eb7baee169e7f3a28 P 8571114a663448e99821d16f67d38bcb: No bootstrap required, opened a new log
I20260621 05:10:25.640635 5847 ts_tablet_manager.cc:1397] T a49392eeeab6454eb7baee169e7f3a28 P 8571114a663448e99821d16f67d38bcb: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260621 05:10:25.641754 5845 raft_consensus.cc:348] T a49392eeeab6454eb7baee169e7f3a28 P 1f4e2cd7408443c0a14c51ebb038ec73 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8571114a663448e99821d16f67d38bcb" member_type: VOTER last_known_addr { host: "127.3.131.131" port: 43851 } } peers { permanent_uuid: "71d95c905a8f4ff181c1de9655d4a593" member_type: VOTER last_known_addr { host: "127.3.131.130" port: 35573 } } peers { permanent_uuid: "1f4e2cd7408443c0a14c51ebb038ec73" member_type: VOTER last_known_addr { host: "127.3.131.129" port: 45955 } }
I20260621 05:10:25.641754 5846 raft_consensus.cc:348] T a49392eeeab6454eb7baee169e7f3a28 P 71d95c905a8f4ff181c1de9655d4a593 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8571114a663448e99821d16f67d38bcb" member_type: VOTER last_known_addr { host: "127.3.131.131" port: 43851 } } peers { permanent_uuid: "71d95c905a8f4ff181c1de9655d4a593" member_type: VOTER last_known_addr { host: "127.3.131.130" port: 35573 } } peers { permanent_uuid: "1f4e2cd7408443c0a14c51ebb038ec73" member_type: VOTER last_known_addr { host: "127.3.131.129" port: 45955 } }
I20260621 05:10:25.641919 5846 raft_consensus.cc:374] T a49392eeeab6454eb7baee169e7f3a28 P 71d95c905a8f4ff181c1de9655d4a593 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 05:10:25.641919 5845 raft_consensus.cc:374] T a49392eeeab6454eb7baee169e7f3a28 P 1f4e2cd7408443c0a14c51ebb038ec73 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 05:10:25.641954 5845 raft_consensus.cc:729] T a49392eeeab6454eb7baee169e7f3a28 P 1f4e2cd7408443c0a14c51ebb038ec73 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f4e2cd7408443c0a14c51ebb038ec73, State: Initialized, Role: FOLLOWER
I20260621 05:10:25.641954 5846 raft_consensus.cc:729] T a49392eeeab6454eb7baee169e7f3a28 P 71d95c905a8f4ff181c1de9655d4a593 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 71d95c905a8f4ff181c1de9655d4a593, State: Initialized, Role: FOLLOWER
I20260621 05:10:25.642081 5846 consensus_queue.cc:260] T a49392eeeab6454eb7baee169e7f3a28 P 71d95c905a8f4ff181c1de9655d4a593 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8571114a663448e99821d16f67d38bcb" member_type: VOTER last_known_addr { host: "127.3.131.131" port: 43851 } } peers { permanent_uuid: "71d95c905a8f4ff181c1de9655d4a593" member_type: VOTER last_known_addr { host: "127.3.131.130" port: 35573 } } peers { permanent_uuid: "1f4e2cd7408443c0a14c51ebb038ec73" member_type: VOTER last_known_addr { host: "127.3.131.129" port: 45955 } }
I20260621 05:10:25.642077 5845 consensus_queue.cc:260] T a49392eeeab6454eb7baee169e7f3a28 P 1f4e2cd7408443c0a14c51ebb038ec73 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8571114a663448e99821d16f67d38bcb" member_type: VOTER last_known_addr { host: "127.3.131.131" port: 43851 } } peers { permanent_uuid: "71d95c905a8f4ff181c1de9655d4a593" member_type: VOTER last_known_addr { host: "127.3.131.130" port: 35573 } } peers { permanent_uuid: "1f4e2cd7408443c0a14c51ebb038ec73" member_type: VOTER last_known_addr { host: "127.3.131.129" port: 45955 } }
I20260621 05:10:25.642338 5845 ts_tablet_manager.cc:1428] T a49392eeeab6454eb7baee169e7f3a28 P 1f4e2cd7408443c0a14c51ebb038ec73: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260621 05:10:25.642338 5846 ts_tablet_manager.cc:1428] T a49392eeeab6454eb7baee169e7f3a28 P 71d95c905a8f4ff181c1de9655d4a593: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260621 05:10:25.642472 5542 heartbeater.cc:503] Master 127.3.131.190:43783 was elected leader, sending a full tablet report...
I20260621 05:10:25.642480 5682 heartbeater.cc:503] Master 127.3.131.190:43783 was elected leader, sending a full tablet report...
I20260621 05:10:25.642531 5847 raft_consensus.cc:348] T a49392eeeab6454eb7baee169e7f3a28 P 8571114a663448e99821d16f67d38bcb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8571114a663448e99821d16f67d38bcb" member_type: VOTER last_known_addr { host: "127.3.131.131" port: 43851 } } peers { permanent_uuid: "71d95c905a8f4ff181c1de9655d4a593" member_type: VOTER last_known_addr { host: "127.3.131.130" port: 35573 } } peers { permanent_uuid: "1f4e2cd7408443c0a14c51ebb038ec73" member_type: VOTER last_known_addr { host: "127.3.131.129" port: 45955 } }
I20260621 05:10:25.642712 5847 raft_consensus.cc:374] T a49392eeeab6454eb7baee169e7f3a28 P 8571114a663448e99821d16f67d38bcb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 05:10:25.642750 5847 raft_consensus.cc:729] T a49392eeeab6454eb7baee169e7f3a28 P 8571114a663448e99821d16f67d38bcb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8571114a663448e99821d16f67d38bcb, State: Initialized, Role: FOLLOWER
I20260621 05:10:25.643270 5847 consensus_queue.cc:260] T a49392eeeab6454eb7baee169e7f3a28 P 8571114a663448e99821d16f67d38bcb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8571114a663448e99821d16f67d38bcb" member_type: VOTER last_known_addr { host: "127.3.131.131" port: 43851 } } peers { permanent_uuid: "71d95c905a8f4ff181c1de9655d4a593" member_type: VOTER last_known_addr { host: "127.3.131.130" port: 35573 } } peers { permanent_uuid: "1f4e2cd7408443c0a14c51ebb038ec73" member_type: VOTER last_known_addr { host: "127.3.131.129" port: 45955 } }
I20260621 05:10:25.643607 5847 ts_tablet_manager.cc:1428] T a49392eeeab6454eb7baee169e7f3a28 P 8571114a663448e99821d16f67d38bcb: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260621 05:10:25.643801 5822 heartbeater.cc:503] Master 127.3.131.190:43783 was elected leader, sending a full tablet report...
I20260621 05:10:25.658393 5852 raft_consensus.cc:482] T a49392eeeab6454eb7baee169e7f3a28 P 1f4e2cd7408443c0a14c51ebb038ec73 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260621 05:10:25.658540 5852 raft_consensus.cc:504] T a49392eeeab6454eb7baee169e7f3a28 P 1f4e2cd7408443c0a14c51ebb038ec73 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8571114a663448e99821d16f67d38bcb" member_type: VOTER last_known_addr { host: "127.3.131.131" port: 43851 } } peers { permanent_uuid: "71d95c905a8f4ff181c1de9655d4a593" member_type: VOTER last_known_addr { host: "127.3.131.130" port: 35573 } } peers { permanent_uuid: "1f4e2cd7408443c0a14c51ebb038ec73" member_type: VOTER last_known_addr { host: "127.3.131.129" port: 45955 } }
W20260621 05:10:25.658829 5683 tablet.cc:2369] T a49392eeeab6454eb7baee169e7f3a28 P 71d95c905a8f4ff181c1de9655d4a593: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260621 05:10:25.658901 5852 leader_election.cc:288] T a49392eeeab6454eb7baee169e7f3a28 P 1f4e2cd7408443c0a14c51ebb038ec73 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8571114a663448e99821d16f67d38bcb (127.3.131.131:43851), 71d95c905a8f4ff181c1de9655d4a593 (127.3.131.130:35573)
I20260621 05:10:25.661743 5842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:10:25.659225 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.131.129:51145 (local address 127.3.131.130:35573)
0621 05:10:25.659283 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0621 05:10:25.659285 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:10:25.659299 (+ 14us) server_negotiation.cc:407] Connection header received
0621 05:10:25.659329 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:10:25.659329 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:10:25.659339 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:10:25.659432 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 05:10:25.659610 (+ 178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:10:25.660097 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:10:25.661010 (+ 913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:10:25.661385 (+ 375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:10:25.661485 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:10:25.661494 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0621 05:10:25.661649 (+ 155us) server_negotiation.cc:299] Negotiation successful
0621 05:10:25.661688 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260621 05:10:25.661743 5435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:10:25.659136 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.131.130:35573 (local address 127.3.131.129:51145)
0621 05:10:25.659216 (+ 80us) negotiation.cc:107] Waiting for socket to connect
0621 05:10:25.659229 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0621 05:10:25.659276 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:10:25.659443 (+ 167us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:10:25.659444 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:10:25.659446 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 05:10:25.659517 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:10:25.659520 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:10:25.660175 (+ 655us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:10:25.660177 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:10:25.660918 (+ 741us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:10:25.660920 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:10:25.661475 (+ 555us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:10:25.661476 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:10:25.661556 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:10:25.661560 (+ 4us) client_negotiation.cc:769] Sending connection context
0621 05:10:25.661631 (+ 71us) client_negotiation.cc:240] Negotiation successful
0621 05:10:25.661669 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":52}
I20260621 05:10:25.661840 5841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:10:25.659424 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.131.129:43667 (local address 127.3.131.131:43851)
0621 05:10:25.659468 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0621 05:10:25.659470 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:10:25.659477 (+ 7us) server_negotiation.cc:407] Connection header received
0621 05:10:25.659561 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:10:25.659562 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:10:25.659571 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:10:25.659628 (+ 57us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 05:10:25.659800 (+ 172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:10:25.660262 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:10:25.661136 (+ 874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:10:25.661504 (+ 368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:10:25.661587 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:10:25.661595 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0621 05:10:25.661743 (+ 148us) server_negotiation.cc:299] Negotiation successful
0621 05:10:25.661778 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260621 05:10:25.661855 5854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:10:25.659160 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.131.131:43851 (local address 127.3.131.129:43667)
0621 05:10:25.659374 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0621 05:10:25.659382 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0621 05:10:25.659419 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:10:25.659634 (+ 215us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:10:25.659634 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:10:25.659636 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 05:10:25.659723 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:10:25.659725 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:10:25.660344 (+ 619us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:10:25.660345 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:10:25.661048 (+ 703us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:10:25.661050 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:10:25.661587 (+ 537us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:10:25.661588 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:10:25.661667 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:10:25.661670 (+ 3us) client_negotiation.cc:769] Sending connection context
0621 05:10:25.661736 (+ 66us) client_negotiation.cc:240] Negotiation successful
0621 05:10:25.661773 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":199,"thread_start_us":136,"threads_started":1}
I20260621 05:10:25.662103 5636 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a49392eeeab6454eb7baee169e7f3a28" candidate_uuid: "1f4e2cd7408443c0a14c51ebb038ec73" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "71d95c905a8f4ff181c1de9655d4a593" is_pre_election: true
I20260621 05:10:25.662216 5776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a49392eeeab6454eb7baee169e7f3a28" candidate_uuid: "1f4e2cd7408443c0a14c51ebb038ec73" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8571114a663448e99821d16f67d38bcb" is_pre_election: true
I20260621 05:10:25.662271 5636 raft_consensus.cc:2454] T a49392eeeab6454eb7baee169e7f3a28 P 71d95c905a8f4ff181c1de9655d4a593 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1f4e2cd7408443c0a14c51ebb038ec73 in term 0.
I20260621 05:10:25.662335 5776 raft_consensus.cc:2454] T a49392eeeab6454eb7baee169e7f3a28 P 8571114a663448e99821d16f67d38bcb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1f4e2cd7408443c0a14c51ebb038ec73 in term 0.
I20260621 05:10:25.662549 5429 leader_election.cc:302] T a49392eeeab6454eb7baee169e7f3a28 P 1f4e2cd7408443c0a14c51ebb038ec73 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f4e2cd7408443c0a14c51ebb038ec73, 8571114a663448e99821d16f67d38bcb; no voters:
I20260621 05:10:25.662678 5852 raft_consensus.cc:2781] T a49392eeeab6454eb7baee169e7f3a28 P 1f4e2cd7408443c0a14c51ebb038ec73 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260621 05:10:25.662751 5852 raft_consensus.cc:482] T a49392eeeab6454eb7baee169e7f3a28 P 1f4e2cd7408443c0a14c51ebb038ec73 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260621 05:10:25.662770 5852 raft_consensus.cc:3037] T a49392eeeab6454eb7baee169e7f3a28 P 1f4e2cd7408443c0a14c51ebb038ec73 [term 0 FOLLOWER]: Advancing to term 1
I20260621 05:10:25.663565 5852 raft_consensus.cc:504] T a49392eeeab6454eb7baee169e7f3a28 P 1f4e2cd7408443c0a14c51ebb038ec73 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8571114a663448e99821d16f67d38bcb" member_type: VOTER last_known_addr { host: "127.3.131.131" port: 43851 } } peers { permanent_uuid: "71d95c905a8f4ff181c1de9655d4a593" member_type: VOTER last_known_addr { host: "127.3.131.130" port: 35573 } } peers { permanent_uuid: "1f4e2cd7408443c0a14c51ebb038ec73" member_type: VOTER last_known_addr { host: "127.3.131.129" port: 45955 } }
I20260621 05:10:25.663743 5852 leader_election.cc:288] T a49392eeeab6454eb7baee169e7f3a28 P 1f4e2cd7408443c0a14c51ebb038ec73 [CANDIDATE]: Term 1 election: Requested vote from peers 8571114a663448e99821d16f67d38bcb (127.3.131.131:43851), 71d95c905a8f4ff181c1de9655d4a593 (127.3.131.130:35573)
I20260621 05:10:25.663853 5776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a49392eeeab6454eb7baee169e7f3a28" candidate_uuid: "1f4e2cd7408443c0a14c51ebb038ec73" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8571114a663448e99821d16f67d38bcb"
I20260621 05:10:25.663921 5636 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a49392eeeab6454eb7baee169e7f3a28" candidate_uuid: "1f4e2cd7408443c0a14c51ebb038ec73" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "71d95c905a8f4ff181c1de9655d4a593"
I20260621 05:10:25.663942 5776 raft_consensus.cc:3037] T a49392eeeab6454eb7baee169e7f3a28 P 8571114a663448e99821d16f67d38bcb [term 0 FOLLOWER]: Advancing to term 1
I20260621 05:10:25.663980 5636 raft_consensus.cc:3037] T a49392eeeab6454eb7baee169e7f3a28 P 71d95c905a8f4ff181c1de9655d4a593 [term 0 FOLLOWER]: Advancing to term 1
I20260621 05:10:25.664598 5776 raft_consensus.cc:2454] T a49392eeeab6454eb7baee169e7f3a28 P 8571114a663448e99821d16f67d38bcb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1f4e2cd7408443c0a14c51ebb038ec73 in term 1.
I20260621 05:10:25.664602 5636 raft_consensus.cc:2454] T a49392eeeab6454eb7baee169e7f3a28 P 71d95c905a8f4ff181c1de9655d4a593 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1f4e2cd7408443c0a14c51ebb038ec73 in term 1.
I20260621 05:10:25.665158 5428 leader_election.cc:302] T a49392eeeab6454eb7baee169e7f3a28 P 1f4e2cd7408443c0a14c51ebb038ec73 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f4e2cd7408443c0a14c51ebb038ec73, 71d95c905a8f4ff181c1de9655d4a593; no voters:
I20260621 05:10:25.665320 5852 raft_consensus.cc:2781] T a49392eeeab6454eb7baee169e7f3a28 P 1f4e2cd7408443c0a14c51ebb038ec73 [term 1 FOLLOWER]: Leader election won for term 1
I20260621 05:10:25.665509 5852 raft_consensus.cc:686] T a49392eeeab6454eb7baee169e7f3a28 P 1f4e2cd7408443c0a14c51ebb038ec73 [term 1 LEADER]: Becoming Leader. State: Replica: 1f4e2cd7408443c0a14c51ebb038ec73, State: Running, Role: LEADER
I20260621 05:10:25.665613 5852 consensus_queue.cc:237] T a49392eeeab6454eb7baee169e7f3a28 P 1f4e2cd7408443c0a14c51ebb038ec73 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8571114a663448e99821d16f67d38bcb" member_type: VOTER last_known_addr { host: "127.3.131.131" port: 43851 } } peers { permanent_uuid: "71d95c905a8f4ff181c1de9655d4a593" member_type: VOTER last_known_addr { host: "127.3.131.130" port: 35573 } } peers { permanent_uuid: "1f4e2cd7408443c0a14c51ebb038ec73" member_type: VOTER last_known_addr { host: "127.3.131.129" port: 45955 } }
I20260621 05:10:25.666503 5338 catalog_manager.cc:5495] T a49392eeeab6454eb7baee169e7f3a28 P 1f4e2cd7408443c0a14c51ebb038ec73 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1f4e2cd7408443c0a14c51ebb038ec73 (127.3.131.129). New cstate: current_term: 1 leader_uuid: "1f4e2cd7408443c0a14c51ebb038ec73" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8571114a663448e99821d16f67d38bcb" member_type: VOTER last_known_addr { host: "127.3.131.131" port: 43851 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "71d95c905a8f4ff181c1de9655d4a593" member_type: VOTER last_known_addr { host: "127.3.131.130" port: 35573 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1f4e2cd7408443c0a14c51ebb038ec73" member_type: VOTER last_known_addr { host: "127.3.131.129" port: 45955 } health_report { overall_health: HEALTHY } } }
Jun 21 05:10:25 dist-test-slave-vrq1 krb5kdc[5294](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782018625, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.131.190@KRBTEST.COM for kudu/127.3.131.190@KRBTEST.COM
I20260621 05:10:25.699105 5862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:10:25.691980 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.131.190:43783 (local address 127.0.0.1:39502)
0621 05:10:25.692273 (+ 293us) negotiation.cc:107] Waiting for socket to connect
0621 05:10:25.692283 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0621 05:10:25.692379 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:10:25.692551 (+ 172us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:10:25.692552 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:10:25.692894 (+ 342us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:10:25.692950 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:10:25.692953 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:10:25.693604 (+ 651us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:10:25.693605 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:10:25.694128 (+ 523us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:10:25.694129 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:10:25.694505 (+ 376us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:10:25.694506 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:10:25.694560 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:10:25.695092 (+ 532us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 05:10:25.695096 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:10:25.696586 (+ 1490us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 05:10:25.698129 (+ 1543us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:10:25.698136 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:10:25.698138 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:10:25.698348 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:10:25.698545 (+ 197us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:10:25.698546 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:10:25.698546 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:10:25.698603 (+ 57us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:10:25.698837 (+ 234us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 05:10:25.698843 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 05:10:25.698887 (+ 44us) client_negotiation.cc:769] Sending connection context
0621 05:10:25.698942 (+ 55us) client_negotiation.cc:240] Negotiation successful
0621 05:10:25.699027 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":267,"thread_start_us":177,"threads_started":1}
I20260621 05:10:25.699330 5389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:10:25.692059 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39502 (local address 127.3.131.190:43783)
0621 05:10:25.692138 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0621 05:10:25.692145 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0621 05:10:25.692359 (+ 214us) server_negotiation.cc:407] Connection header received
0621 05:10:25.692465 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:10:25.692465 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:10:25.692475 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:10:25.692545 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:10:25.693038 (+ 493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:10:25.693522 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:10:25.694237 (+ 715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:10:25.694421 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:10:25.694505 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:10:25.696668 (+ 2163us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:10:25.696669 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:10:25.696669 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:10:25.696676 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:10:25.698010 (+ 1334us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:10:25.698426 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:10:25.698427 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:10:25.698427 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:10:25.698480 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:10:25.698656 (+ 176us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:10:25.698656 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:10:25.698657 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:10:25.698782 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:10:25.698833 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0621 05:10:25.699037 (+ 204us) server_negotiation.cc:299] Negotiation successful
0621 05:10:25.699117 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260621 05:10:25.701555 5338 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.3.131.190@KRBTEST.COM'} at 127.0.0.1:39502:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260621 05:10:25.701864 5338 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260621 05:10:25.704070 5756 tablet_service.cc:1467] Processing CreateTablet for tablet 9304a9b94d2444dcb274278f840c5fe1 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=40360a38557d4f489f7fceeb50a82100]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260621 05:10:25.704049 5476 tablet_service.cc:1467] Processing CreateTablet for tablet 9304a9b94d2444dcb274278f840c5fe1 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=40360a38557d4f489f7fceeb50a82100]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260621 05:10:25.704051 5616 tablet_service.cc:1467] Processing CreateTablet for tablet 9304a9b94d2444dcb274278f840c5fe1 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=40360a38557d4f489f7fceeb50a82100]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260621 05:10:25.704257 5616 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9304a9b94d2444dcb274278f840c5fe1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 05:10:25.704257 5476 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9304a9b94d2444dcb274278f840c5fe1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 05:10:25.704260 5756 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9304a9b94d2444dcb274278f840c5fe1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 05:10:25.705592 5847 tablet_bootstrap.cc:492] T 9304a9b94d2444dcb274278f840c5fe1 P 8571114a663448e99821d16f67d38bcb: Bootstrap starting.
I20260621 05:10:25.705801 5846 tablet_bootstrap.cc:492] T 9304a9b94d2444dcb274278f840c5fe1 P 71d95c905a8f4ff181c1de9655d4a593: Bootstrap starting.
I20260621 05:10:25.706241 5847 tablet_bootstrap.cc:654] T 9304a9b94d2444dcb274278f840c5fe1 P 8571114a663448e99821d16f67d38bcb: Neither blocks nor log segments found. Creating new log.
I20260621 05:10:25.706555 5846 tablet_bootstrap.cc:654] T 9304a9b94d2444dcb274278f840c5fe1 P 71d95c905a8f4ff181c1de9655d4a593: Neither blocks nor log segments found. Creating new log.
I20260621 05:10:25.706905 5845 tablet_bootstrap.cc:492] T 9304a9b94d2444dcb274278f840c5fe1 P 1f4e2cd7408443c0a14c51ebb038ec73: Bootstrap starting.
I20260621 05:10:25.707132 5847 tablet_bootstrap.cc:492] T 9304a9b94d2444dcb274278f840c5fe1 P 8571114a663448e99821d16f67d38bcb: No bootstrap required, opened a new log
I20260621 05:10:25.707209 5847 ts_tablet_manager.cc:1397] T 9304a9b94d2444dcb274278f840c5fe1 P 8571114a663448e99821d16f67d38bcb: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260621 05:10:25.707263 5846 tablet_bootstrap.cc:492] T 9304a9b94d2444dcb274278f840c5fe1 P 71d95c905a8f4ff181c1de9655d4a593: No bootstrap required, opened a new log
I20260621 05:10:25.707312 5846 ts_tablet_manager.cc:1397] T 9304a9b94d2444dcb274278f840c5fe1 P 71d95c905a8f4ff181c1de9655d4a593: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260621 05:10:25.707391 5845 tablet_bootstrap.cc:654] T 9304a9b94d2444dcb274278f840c5fe1 P 1f4e2cd7408443c0a14c51ebb038ec73: Neither blocks nor log segments found. Creating new log.
I20260621 05:10:25.707365 5847 raft_consensus.cc:348] T 9304a9b94d2444dcb274278f840c5fe1 P 8571114a663448e99821d16f67d38bcb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f4e2cd7408443c0a14c51ebb038ec73" member_type: VOTER last_known_addr { host: "127.3.131.129" port: 45955 } } peers { permanent_uuid: "71d95c905a8f4ff181c1de9655d4a593" member_type: VOTER last_known_addr { host: "127.3.131.130" port: 35573 } } peers { permanent_uuid: "8571114a663448e99821d16f67d38bcb" member_type: VOTER last_known_addr { host: "127.3.131.131" port: 43851 } }
I20260621 05:10:25.707442 5847 raft_consensus.cc:374] T 9304a9b94d2444dcb274278f840c5fe1 P 8571114a663448e99821d16f67d38bcb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 05:10:25.707479 5847 raft_consensus.cc:729] T 9304a9b94d2444dcb274278f840c5fe1 P 8571114a663448e99821d16f67d38bcb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8571114a663448e99821d16f67d38bcb, State: Initialized, Role: FOLLOWER
I20260621 05:10:25.707459 5846 raft_consensus.cc:348] T 9304a9b94d2444dcb274278f840c5fe1 P 71d95c905a8f4ff181c1de9655d4a593 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f4e2cd7408443c0a14c51ebb038ec73" member_type: VOTER last_known_addr { host: "127.3.131.129" port: 45955 } } peers { permanent_uuid: "71d95c905a8f4ff181c1de9655d4a593" member_type: VOTER last_known_addr { host: "127.3.131.130" port: 35573 } } peers { permanent_uuid: "8571114a663448e99821d16f67d38bcb" member_type: VOTER last_known_addr { host: "127.3.131.131" port: 43851 } }
I20260621 05:10:25.707561 5846 raft_consensus.cc:374] T 9304a9b94d2444dcb274278f840c5fe1 P 71d95c905a8f4ff181c1de9655d4a593 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 05:10:25.707540 5847 consensus_queue.cc:260] T 9304a9b94d2444dcb274278f840c5fe1 P 8571114a663448e99821d16f67d38bcb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f4e2cd7408443c0a14c51ebb038ec73" member_type: VOTER last_known_addr { host: "127.3.131.129" port: 45955 } } peers { permanent_uuid: "71d95c905a8f4ff181c1de9655d4a593" member_type: VOTER last_known_addr { host: "127.3.131.130" port: 35573 } } peers { permanent_uuid: "8571114a663448e99821d16f67d38bcb" member_type: VOTER last_known_addr { host: "127.3.131.131" port: 43851 } }
I20260621 05:10:25.707589 5846 raft_consensus.cc:729] T 9304a9b94d2444dcb274278f840c5fe1 P 71d95c905a8f4ff181c1de9655d4a593 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 71d95c905a8f4ff181c1de9655d4a593, State: Initialized, Role: FOLLOWER
I20260621 05:10:25.707647 5846 consensus_queue.cc:260] T 9304a9b94d2444dcb274278f840c5fe1 P 71d95c905a8f4ff181c1de9655d4a593 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f4e2cd7408443c0a14c51ebb038ec73" member_type: VOTER last_known_addr { host: "127.3.131.129" port: 45955 } } peers { permanent_uuid: "71d95c905a8f4ff181c1de9655d4a593" member_type: VOTER last_known_addr { host: "127.3.131.130" port: 35573 } } peers { permanent_uuid: "8571114a663448e99821d16f67d38bcb" member_type: VOTER last_known_addr { host: "127.3.131.131" port: 43851 } }
I20260621 05:10:25.707669 5847 ts_tablet_manager.cc:1428] T 9304a9b94d2444dcb274278f840c5fe1 P 8571114a663448e99821d16f67d38bcb: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260621 05:10:25.707738 5846 ts_tablet_manager.cc:1428] T 9304a9b94d2444dcb274278f840c5fe1 P 71d95c905a8f4ff181c1de9655d4a593: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260621 05:10:25.707849 5853 tablet_replica.cc:439] T 9304a9b94d2444dcb274278f840c5fe1 P 8571114a663448e99821d16f67d38bcb: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f4e2cd7408443c0a14c51ebb038ec73" member_type: VOTER last_known_addr { host: "127.3.131.129" port: 45955 } } peers { permanent_uuid: "71d95c905a8f4ff181c1de9655d4a593" member_type: VOTER last_known_addr { host: "127.3.131.130" port: 35573 } } peers { permanent_uuid: "8571114a663448e99821d16f67d38bcb" member_type: VOTER last_known_addr { host: "127.3.131.131" port: 43851 } } }
I20260621 05:10:25.707901 5851 tablet_replica.cc:439] T 9304a9b94d2444dcb274278f840c5fe1 P 71d95c905a8f4ff181c1de9655d4a593: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f4e2cd7408443c0a14c51ebb038ec73" member_type: VOTER last_known_addr { host: "127.3.131.129" port: 45955 } } peers { permanent_uuid: "71d95c905a8f4ff181c1de9655d4a593" member_type: VOTER last_known_addr { host: "127.3.131.130" port: 35573 } } peers { permanent_uuid: "8571114a663448e99821d16f67d38bcb" member_type: VOTER last_known_addr { host: "127.3.131.131" port: 43851 } } }
I20260621 05:10:25.707978 5851 tablet_replica.cc:442] T 9304a9b94d2444dcb274278f840c5fe1 P 71d95c905a8f4ff181c1de9655d4a593: This TxnStatusTablet replica's current role is: FOLLOWER
I20260621 05:10:25.707969 5853 tablet_replica.cc:442] T 9304a9b94d2444dcb274278f840c5fe1 P 8571114a663448e99821d16f67d38bcb: This TxnStatusTablet replica's current role is: FOLLOWER
I20260621 05:10:25.708526 5845 tablet_bootstrap.cc:492] T 9304a9b94d2444dcb274278f840c5fe1 P 1f4e2cd7408443c0a14c51ebb038ec73: No bootstrap required, opened a new log
I20260621 05:10:25.708580 5845 ts_tablet_manager.cc:1397] T 9304a9b94d2444dcb274278f840c5fe1 P 1f4e2cd7408443c0a14c51ebb038ec73: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260621 05:10:25.708756 5845 raft_consensus.cc:348] T 9304a9b94d2444dcb274278f840c5fe1 P 1f4e2cd7408443c0a14c51ebb038ec73 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f4e2cd7408443c0a14c51ebb038ec73" member_type: VOTER last_known_addr { host: "127.3.131.129" port: 45955 } } peers { permanent_uuid: "71d95c905a8f4ff181c1de9655d4a593" member_type: VOTER last_known_addr { host: "127.3.131.130" port: 35573 } } peers { permanent_uuid: "8571114a663448e99821d16f67d38bcb" member_type: VOTER last_known_addr { host: "127.3.131.131" port: 43851 } }
I20260621 05:10:25.708839 5845 raft_consensus.cc:374] T 9304a9b94d2444dcb274278f840c5fe1 P 1f4e2cd7408443c0a14c51ebb038ec73 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 05:10:25.708855 5845 raft_consensus.cc:729] T 9304a9b94d2444dcb274278f840c5fe1 P 1f4e2cd7408443c0a14c51ebb038ec73 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f4e2cd7408443c0a14c51ebb038ec73, State: Initialized, Role: FOLLOWER
I20260621 05:10:25.708889 5845 consensus_queue.cc:260] T 9304a9b94d2444dcb274278f840c5fe1 P 1f4e2cd7408443c0a14c51ebb038ec73 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f4e2cd7408443c0a14c51ebb038ec73" member_type: VOTER last_known_addr { host: "127.3.131.129" port: 45955 } } peers { permanent_uuid: "71d95c905a8f4ff181c1de9655d4a593" member_type: VOTER last_known_addr { host: "127.3.131.130" port: 35573 } } peers { permanent_uuid: "8571114a663448e99821d16f67d38bcb" member_type: VOTER last_known_addr { host: "127.3.131.131" port: 43851 } }
I20260621 05:10:25.708976 5845 ts_tablet_manager.cc:1428] T 9304a9b94d2444dcb274278f840c5fe1 P 1f4e2cd7408443c0a14c51ebb038ec73: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260621 05:10:25.709098 5852 tablet_replica.cc:439] T 9304a9b94d2444dcb274278f840c5fe1 P 1f4e2cd7408443c0a14c51ebb038ec73: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f4e2cd7408443c0a14c51ebb038ec73" member_type: VOTER last_known_addr { host: "127.3.131.129" port: 45955 } } peers { permanent_uuid: "71d95c905a8f4ff181c1de9655d4a593" member_type: VOTER last_known_addr { host: "127.3.131.130" port: 35573 } } peers { permanent_uuid: "8571114a663448e99821d16f67d38bcb" member_type: VOTER last_known_addr { host: "127.3.131.131" port: 43851 } } }
I20260621 05:10:25.709179 5852 tablet_replica.cc:442] T 9304a9b94d2444dcb274278f840c5fe1 P 1f4e2cd7408443c0a14c51ebb038ec73: This TxnStatusTablet replica's current role is: FOLLOWER
W20260621 05:10:25.738920 5543 tablet.cc:2369] T 9304a9b94d2444dcb274278f840c5fe1 P 1f4e2cd7408443c0a14c51ebb038ec73: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260621 05:10:25.833340 5823 tablet.cc:2369] T 9304a9b94d2444dcb274278f840c5fe1 P 8571114a663448e99821d16f67d38bcb: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260621 05:10:25.947685 5851 raft_consensus.cc:482] T 9304a9b94d2444dcb274278f840c5fe1 P 71d95c905a8f4ff181c1de9655d4a593 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260621 05:10:25.947793 5851 raft_consensus.cc:504] T 9304a9b94d2444dcb274278f840c5fe1 P 71d95c905a8f4ff181c1de9655d4a593 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f4e2cd7408443c0a14c51ebb038ec73" member_type: VOTER last_known_addr { host: "127.3.131.129" port: 45955 } } peers { permanent_uuid: "71d95c905a8f4ff181c1de9655d4a593" member_type: VOTER last_known_addr { host: "127.3.131.130" port: 35573 } } peers { permanent_uuid: "8571114a663448e99821d16f67d38bcb" member_type: VOTER last_known_addr { host: "127.3.131.131" port: 43851 } }
I20260621 05:10:25.948110 5851 leader_election.cc:288] T 9304a9b94d2444dcb274278f840c5fe1 P 71d95c905a8f4ff181c1de9655d4a593 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1f4e2cd7408443c0a14c51ebb038ec73 (127.3.131.129:45955), 8571114a663448e99821d16f67d38bcb (127.3.131.131:43851)
I20260621 05:10:25.951475 5841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:10:25.948824 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.131.130:33251 (local address 127.3.131.131:43851)
0621 05:10:25.948888 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0621 05:10:25.948890 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:10:25.948900 (+ 10us) server_negotiation.cc:407] Connection header received
0621 05:10:25.948963 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:10:25.948964 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:10:25.948972 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:10:25.949022 (+ 50us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 05:10:25.949199 (+ 177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:10:25.949652 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:10:25.950699 (+ 1047us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:10:25.951086 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:10:25.951172 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:10:25.951181 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0621 05:10:25.951359 (+ 178us) server_negotiation.cc:299] Negotiation successful
0621 05:10:25.951400 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260621 05:10:25.951493 5867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:10:25.948727 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.131.131:43851 (local address 127.3.131.130:33251)
0621 05:10:25.948821 (+ 94us) negotiation.cc:107] Waiting for socket to connect
0621 05:10:25.948832 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0621 05:10:25.948900 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:10:25.949033 (+ 133us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:10:25.949033 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:10:25.949035 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 05:10:25.949122 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:10:25.949124 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:10:25.949759 (+ 635us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:10:25.949760 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:10:25.950599 (+ 839us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:10:25.950601 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:10:25.951202 (+ 601us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:10:25.951203 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:10:25.951273 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:10:25.951277 (+ 4us) client_negotiation.cc:769] Sending connection context
0621 05:10:25.951339 (+ 62us) client_negotiation.cc:240] Negotiation successful
0621 05:10:25.951379 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":78,"mutex_wait_us":46,"thread_start_us":47,"threads_started":1}
I20260621 05:10:25.951500 5844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:10:25.948495 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.131.130:52299 (local address 127.3.131.129:45955)
0621 05:10:25.948591 (+ 96us) server_negotiation.cc:206] Beginning negotiation
0621 05:10:25.948593 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:10:25.948702 (+ 109us) server_negotiation.cc:407] Connection header received
0621 05:10:25.948817 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:10:25.948818 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:10:25.948830 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:10:25.948865 (+ 35us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 05:10:25.949116 (+ 251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:10:25.949637 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:10:25.950747 (+ 1110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:10:25.951113 (+ 366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:10:25.951236 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:10:25.951244 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0621 05:10:25.951391 (+ 147us) server_negotiation.cc:299] Negotiation successful
0621 05:10:25.951433 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260621 05:10:25.951519 5866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:10:25.948418 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.131.129:45955 (local address 127.3.131.130:52299)
0621 05:10:25.948612 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0621 05:10:25.948626 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0621 05:10:25.948706 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:10:25.948960 (+ 254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:10:25.948961 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:10:25.948963 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 05:10:25.949027 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:10:25.949030 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:10:25.949723 (+ 693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:10:25.949724 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:10:25.950644 (+ 920us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:10:25.950647 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:10:25.951213 (+ 566us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:10:25.951214 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:10:25.951300 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:10:25.951304 (+ 4us) client_negotiation.cc:769] Sending connection context
0621 05:10:25.951379 (+ 75us) client_negotiation.cc:240] Negotiation successful
0621 05:10:25.951422 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":175,"mutex_wait_us":72,"thread_start_us":149,"threads_started":1}
I20260621 05:10:25.951758 5776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9304a9b94d2444dcb274278f840c5fe1" candidate_uuid: "71d95c905a8f4ff181c1de9655d4a593" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8571114a663448e99821d16f67d38bcb" is_pre_election: true
I20260621 05:10:25.951860 5776 raft_consensus.cc:2454] T 9304a9b94d2444dcb274278f840c5fe1 P 8571114a663448e99821d16f67d38bcb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 71d95c905a8f4ff181c1de9655d4a593 in term 0.
I20260621 05:10:25.951867 5496 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9304a9b94d2444dcb274278f840c5fe1" candidate_uuid: "71d95c905a8f4ff181c1de9655d4a593" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f4e2cd7408443c0a14c51ebb038ec73" is_pre_election: true
I20260621 05:10:25.951989 5496 raft_consensus.cc:2454] T 9304a9b94d2444dcb274278f840c5fe1 P 1f4e2cd7408443c0a14c51ebb038ec73 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 71d95c905a8f4ff181c1de9655d4a593 in term 0.
I20260621 05:10:25.952052 5569 leader_election.cc:302] T 9304a9b94d2444dcb274278f840c5fe1 P 71d95c905a8f4ff181c1de9655d4a593 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 71d95c905a8f4ff181c1de9655d4a593, 8571114a663448e99821d16f67d38bcb; no voters:
I20260621 05:10:25.952193 5851 raft_consensus.cc:2781] T 9304a9b94d2444dcb274278f840c5fe1 P 71d95c905a8f4ff181c1de9655d4a593 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260621 05:10:25.952248 5851 raft_consensus.cc:482] T 9304a9b94d2444dcb274278f840c5fe1 P 71d95c905a8f4ff181c1de9655d4a593 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260621 05:10:25.952265 5851 raft_consensus.cc:3037] T 9304a9b94d2444dcb274278f840c5fe1 P 71d95c905a8f4ff181c1de9655d4a593 [term 0 FOLLOWER]: Advancing to term 1
I20260621 05:10:25.952996 5851 raft_consensus.cc:504] T 9304a9b94d2444dcb274278f840c5fe1 P 71d95c905a8f4ff181c1de9655d4a593 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f4e2cd7408443c0a14c51ebb038ec73" member_type: VOTER last_known_addr { host: "127.3.131.129" port: 45955 } } peers { permanent_uuid: "71d95c905a8f4ff181c1de9655d4a593" member_type: VOTER last_known_addr { host: "127.3.131.130" port: 35573 } } peers { permanent_uuid: "8571114a663448e99821d16f67d38bcb" member_type: VOTER last_known_addr { host: "127.3.131.131" port: 43851 } }
I20260621 05:10:25.953101 5851 leader_election.cc:288] T 9304a9b94d2444dcb274278f840c5fe1 P 71d95c905a8f4ff181c1de9655d4a593 [CANDIDATE]: Term 1 election: Requested vote from peers 1f4e2cd7408443c0a14c51ebb038ec73 (127.3.131.129:45955), 8571114a663448e99821d16f67d38bcb (127.3.131.131:43851)
I20260621 05:10:25.953392 5776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9304a9b94d2444dcb274278f840c5fe1" candidate_uuid: "71d95c905a8f4ff181c1de9655d4a593" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8571114a663448e99821d16f67d38bcb"
I20260621 05:10:25.953380 5496 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9304a9b94d2444dcb274278f840c5fe1" candidate_uuid: "71d95c905a8f4ff181c1de9655d4a593" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f4e2cd7408443c0a14c51ebb038ec73"
I20260621 05:10:25.953475 5496 raft_consensus.cc:3037] T 9304a9b94d2444dcb274278f840c5fe1 P 1f4e2cd7408443c0a14c51ebb038ec73 [term 0 FOLLOWER]: Advancing to term 1
I20260621 05:10:25.953474 5776 raft_consensus.cc:3037] T 9304a9b94d2444dcb274278f840c5fe1 P 8571114a663448e99821d16f67d38bcb [term 0 FOLLOWER]: Advancing to term 1
I20260621 05:10:25.954324 5496 raft_consensus.cc:2454] T 9304a9b94d2444dcb274278f840c5fe1 P 1f4e2cd7408443c0a14c51ebb038ec73 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 71d95c905a8f4ff181c1de9655d4a593 in term 1.
I20260621 05:10:25.954324 5776 raft_consensus.cc:2454] T 9304a9b94d2444dcb274278f840c5fe1 P 8571114a663448e99821d16f67d38bcb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 71d95c905a8f4ff181c1de9655d4a593 in term 1.
I20260621 05:10:25.954506 5569 leader_election.cc:302] T 9304a9b94d2444dcb274278f840c5fe1 P 71d95c905a8f4ff181c1de9655d4a593 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 71d95c905a8f4ff181c1de9655d4a593, 8571114a663448e99821d16f67d38bcb; no voters:
I20260621 05:10:25.954623 5851 raft_consensus.cc:2781] T 9304a9b94d2444dcb274278f840c5fe1 P 71d95c905a8f4ff181c1de9655d4a593 [term 1 FOLLOWER]: Leader election won for term 1
I20260621 05:10:25.954758 5851 raft_consensus.cc:686] T 9304a9b94d2444dcb274278f840c5fe1 P 71d95c905a8f4ff181c1de9655d4a593 [term 1 LEADER]: Becoming Leader. State: Replica: 71d95c905a8f4ff181c1de9655d4a593, State: Running, Role: LEADER
I20260621 05:10:25.954865 5851 consensus_queue.cc:237] T 9304a9b94d2444dcb274278f840c5fe1 P 71d95c905a8f4ff181c1de9655d4a593 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f4e2cd7408443c0a14c51ebb038ec73" member_type: VOTER last_known_addr { host: "127.3.131.129" port: 45955 } } peers { permanent_uuid: "71d95c905a8f4ff181c1de9655d4a593" member_type: VOTER last_known_addr { host: "127.3.131.130" port: 35573 } } peers { permanent_uuid: "8571114a663448e99821d16f67d38bcb" member_type: VOTER last_known_addr { host: "127.3.131.131" port: 43851 } }
I20260621 05:10:25.955328 5868 tablet_replica.cc:439] T 9304a9b94d2444dcb274278f840c5fe1 P 71d95c905a8f4ff181c1de9655d4a593: TxnStatusTablet state changed. Reason: New leader 71d95c905a8f4ff181c1de9655d4a593. Latest consensus state: current_term: 1 leader_uuid: "71d95c905a8f4ff181c1de9655d4a593" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f4e2cd7408443c0a14c51ebb038ec73" member_type: VOTER last_known_addr { host: "127.3.131.129" port: 45955 } } peers { permanent_uuid: "71d95c905a8f4ff181c1de9655d4a593" member_type: VOTER last_known_addr { host: "127.3.131.130" port: 35573 } } peers { permanent_uuid: "8571114a663448e99821d16f67d38bcb" member_type: VOTER last_known_addr { host: "127.3.131.131" port: 43851 } } }
I20260621 05:10:25.955418 5868 tablet_replica.cc:442] T 9304a9b94d2444dcb274278f840c5fe1 P 71d95c905a8f4ff181c1de9655d4a593: This TxnStatusTablet replica's current role is: LEADER
I20260621 05:10:25.955592 5870 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260621 05:10:25.955713 5870 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260621 05:10:25.955816 5338 catalog_manager.cc:5495] T 9304a9b94d2444dcb274278f840c5fe1 P 71d95c905a8f4ff181c1de9655d4a593 reported cstate change: term changed from 0 to 1, leader changed from <none> to 71d95c905a8f4ff181c1de9655d4a593 (127.3.131.130). New cstate: current_term: 1 leader_uuid: "71d95c905a8f4ff181c1de9655d4a593" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f4e2cd7408443c0a14c51ebb038ec73" member_type: VOTER last_known_addr { host: "127.3.131.129" port: 45955 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "71d95c905a8f4ff181c1de9655d4a593" member_type: VOTER last_known_addr { host: "127.3.131.130" port: 35573 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8571114a663448e99821d16f67d38bcb" member_type: VOTER last_known_addr { host: "127.3.131.131" port: 43851 } health_report { overall_health: UNKNOWN } } }
I20260621 05:10:25.995306 5842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:10:25.992513 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48480 (local address 127.3.131.130:35573)
0621 05:10:25.992570 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0621 05:10:25.992573 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0621 05:10:25.992591 (+ 18us) server_negotiation.cc:407] Connection header received
0621 05:10:25.992711 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:10:25.992712 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:10:25.992735 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:10:25.992827 (+ 92us) server_negotiation.cc:226] Negotiated authn=TOKEN
0621 05:10:25.992974 (+ 147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:10:25.993508 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:10:25.994354 (+ 846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:10:25.994589 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:10:25.994673 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:10:25.994885 (+ 212us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0621 05:10:25.995001 (+ 116us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0621 05:10:25.995100 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0621 05:10:25.995162 (+ 62us) server_negotiation.cc:299] Negotiation successful
0621 05:10:25.995216 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260621 05:10:25.995311 5862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:10:25.992438 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.131.130:35573 (local address 127.0.0.1:48480)
0621 05:10:25.992523 (+ 85us) negotiation.cc:107] Waiting for socket to connect
0621 05:10:25.992536 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0621 05:10:25.992594 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:10:25.992833 (+ 239us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:10:25.992834 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:10:25.992835 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0621 05:10:25.992890 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:10:25.992895 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:10:25.993597 (+ 702us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:10:25.993598 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:10:25.994257 (+ 659us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:10:25.994259 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:10:25.994672 (+ 413us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:10:25.994673 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:10:25.994754 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:10:25.994769 (+ 15us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0621 05:10:25.995076 (+ 307us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0621 05:10:25.995092 (+ 16us) client_negotiation.cc:769] Sending connection context
0621 05:10:25.995155 (+ 63us) client_negotiation.cc:240] Negotiation successful
0621 05:10:25.995232 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":44}
I20260621 05:10:25.998071 5776 raft_consensus.cc:1261] T 9304a9b94d2444dcb274278f840c5fe1 P 8571114a663448e99821d16f67d38bcb [term 1 FOLLOWER]: Refusing update from remote peer 71d95c905a8f4ff181c1de9655d4a593: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 05:10:25.998093 5496 raft_consensus.cc:1261] T 9304a9b94d2444dcb274278f840c5fe1 P 1f4e2cd7408443c0a14c51ebb038ec73 [term 1 FOLLOWER]: Refusing update from remote peer 71d95c905a8f4ff181c1de9655d4a593: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 05:10:25.998474 5868 consensus_queue.cc:1035] T 9304a9b94d2444dcb274278f840c5fe1 P 71d95c905a8f4ff181c1de9655d4a593 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8571114a663448e99821d16f67d38bcb" member_type: VOTER last_known_addr { host: "127.3.131.131" port: 43851 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 05:10:25.998636 5851 consensus_queue.cc:1035] T 9304a9b94d2444dcb274278f840c5fe1 P 71d95c905a8f4ff181c1de9655d4a593 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f4e2cd7408443c0a14c51ebb038ec73" member_type: VOTER last_known_addr { host: "127.3.131.129" port: 45955 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 05:10:26.000311 5853 tablet_replica.cc:439] T 9304a9b94d2444dcb274278f840c5fe1 P 8571114a663448e99821d16f67d38bcb: TxnStatusTablet state changed. Reason: New leader 71d95c905a8f4ff181c1de9655d4a593. Latest consensus state: current_term: 1 leader_uuid: "71d95c905a8f4ff181c1de9655d4a593" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f4e2cd7408443c0a14c51ebb038ec73" member_type: VOTER last_known_addr { host: "127.3.131.129" port: 45955 } } peers { permanent_uuid: "71d95c905a8f4ff181c1de9655d4a593" member_type: VOTER last_known_addr { host: "127.3.131.130" port: 35573 } } peers { permanent_uuid: "8571114a663448e99821d16f67d38bcb" member_type: VOTER last_known_addr { host: "127.3.131.131" port: 43851 } } }
I20260621 05:10:26.000418 5853 tablet_replica.cc:442] T 9304a9b94d2444dcb274278f840c5fe1 P 8571114a663448e99821d16f67d38bcb: This TxnStatusTablet replica's current role is: FOLLOWER
I20260621 05:10:26.000856 5852 tablet_replica.cc:439] T 9304a9b94d2444dcb274278f840c5fe1 P 1f4e2cd7408443c0a14c51ebb038ec73: TxnStatusTablet state changed. Reason: New leader 71d95c905a8f4ff181c1de9655d4a593. Latest consensus state: current_term: 1 leader_uuid: "71d95c905a8f4ff181c1de9655d4a593" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f4e2cd7408443c0a14c51ebb038ec73" member_type: VOTER last_known_addr { host: "127.3.131.129" port: 45955 } } peers { permanent_uuid: "71d95c905a8f4ff181c1de9655d4a593" member_type: VOTER last_known_addr { host: "127.3.131.130" port: 35573 } } peers { permanent_uuid: "8571114a663448e99821d16f67d38bcb" member_type: VOTER last_known_addr { host: "127.3.131.131" port: 43851 } } }
I20260621 05:10:26.000953 5852 tablet_replica.cc:442] T 9304a9b94d2444dcb274278f840c5fe1 P 1f4e2cd7408443c0a14c51ebb038ec73: This TxnStatusTablet replica's current role is: FOLLOWER
I20260621 05:10:26.001223 5868 tablet_replica.cc:439] T 9304a9b94d2444dcb274278f840c5fe1 P 71d95c905a8f4ff181c1de9655d4a593: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "71d95c905a8f4ff181c1de9655d4a593" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f4e2cd7408443c0a14c51ebb038ec73" member_type: VOTER last_known_addr { host: "127.3.131.129" port: 45955 } } peers { permanent_uuid: "71d95c905a8f4ff181c1de9655d4a593" member_type: VOTER last_known_addr { host: "127.3.131.130" port: 35573 } } peers { permanent_uuid: "8571114a663448e99821d16f67d38bcb" member_type: VOTER last_known_addr { host: "127.3.131.131" port: 43851 } } }
I20260621 05:10:26.001312 5868 tablet_replica.cc:442] T 9304a9b94d2444dcb274278f840c5fe1 P 71d95c905a8f4ff181c1de9655d4a593: This TxnStatusTablet replica's current role is: LEADER
I20260621 05:10:26.001452 5852 tablet_replica.cc:439] T 9304a9b94d2444dcb274278f840c5fe1 P 1f4e2cd7408443c0a14c51ebb038ec73: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "71d95c905a8f4ff181c1de9655d4a593" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f4e2cd7408443c0a14c51ebb038ec73" member_type: VOTER last_known_addr { host: "127.3.131.129" port: 45955 } } peers { permanent_uuid: "71d95c905a8f4ff181c1de9655d4a593" member_type: VOTER last_known_addr { host: "127.3.131.130" port: 35573 } } peers { permanent_uuid: "8571114a663448e99821d16f67d38bcb" member_type: VOTER last_known_addr { host: "127.3.131.131" port: 43851 } } }
I20260621 05:10:26.001523 5852 tablet_replica.cc:442] T 9304a9b94d2444dcb274278f840c5fe1 P 1f4e2cd7408443c0a14c51ebb038ec73: This TxnStatusTablet replica's current role is: FOLLOWER
I20260621 05:10:26.001632 5874 mvcc.cc:205] Tried to move back new op lower bound from 7299148292085022720 to 7299148291911647232. Current Snapshot: MvccSnapshot[applied={T|T < 7299148292085022720}]
I20260621 05:10:26.001914 5873 mvcc.cc:205] Tried to move back new op lower bound from 7299148292085022720 to 7299148291911647232. Current Snapshot: MvccSnapshot[applied={T|T < 7299148292085022720}]
I20260621 05:10:26.002080 5868 tablet_replica.cc:439] T 9304a9b94d2444dcb274278f840c5fe1 P 71d95c905a8f4ff181c1de9655d4a593: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "71d95c905a8f4ff181c1de9655d4a593" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f4e2cd7408443c0a14c51ebb038ec73" member_type: VOTER last_known_addr { host: "127.3.131.129" port: 45955 } } peers { permanent_uuid: "71d95c905a8f4ff181c1de9655d4a593" member_type: VOTER last_known_addr { host: "127.3.131.130" port: 35573 } } peers { permanent_uuid: "8571114a663448e99821d16f67d38bcb" member_type: VOTER last_known_addr { host: "127.3.131.131" port: 43851 } } }
I20260621 05:10:26.002187 5868 tablet_replica.cc:442] T 9304a9b94d2444dcb274278f840c5fe1 P 71d95c905a8f4ff181c1de9655d4a593: This TxnStatusTablet replica's current role is: LEADER
I20260621 05:10:26.002739 5853 tablet_replica.cc:439] T 9304a9b94d2444dcb274278f840c5fe1 P 8571114a663448e99821d16f67d38bcb: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "71d95c905a8f4ff181c1de9655d4a593" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f4e2cd7408443c0a14c51ebb038ec73" member_type: VOTER last_known_addr { host: "127.3.131.129" port: 45955 } } peers { permanent_uuid: "71d95c905a8f4ff181c1de9655d4a593" member_type: VOTER last_known_addr { host: "127.3.131.130" port: 35573 } } peers { permanent_uuid: "8571114a663448e99821d16f67d38bcb" member_type: VOTER last_known_addr { host: "127.3.131.131" port: 43851 } } }
I20260621 05:10:26.002862 5853 tablet_replica.cc:442] T 9304a9b94d2444dcb274278f840c5fe1 P 8571114a663448e99821d16f67d38bcb: This TxnStatusTablet replica's current role is: FOLLOWER
I20260621 05:10:26.007932 5844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:10:26.004966 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47104 (local address 127.3.131.129:45955)
0621 05:10:26.005006 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0621 05:10:26.005008 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:10:26.005019 (+ 11us) server_negotiation.cc:407] Connection header received
0621 05:10:26.005073 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:10:26.005074 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:10:26.005090 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:10:26.005151 (+ 61us) server_negotiation.cc:226] Negotiated authn=TOKEN
0621 05:10:26.005319 (+ 168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:10:26.005822 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:10:26.006999 (+ 1177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:10:26.007326 (+ 327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:10:26.007390 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:10:26.007616 (+ 226us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0621 05:10:26.007722 (+ 106us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0621 05:10:26.007798 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0621 05:10:26.007860 (+ 62us) server_negotiation.cc:299] Negotiation successful
0621 05:10:26.007880 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260621 05:10:26.013674 5842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:10:26.010845 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.131.129:54613 (local address 127.3.131.130:35573)
0621 05:10:26.010895 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0621 05:10:26.010897 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:10:26.010999 (+ 102us) server_negotiation.cc:407] Connection header received
0621 05:10:26.011043 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:10:26.011044 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:10:26.011053 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:10:26.011100 (+ 47us) server_negotiation.cc:226] Negotiated authn=TOKEN
0621 05:10:26.011323 (+ 223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:10:26.011786 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:10:26.012886 (+ 1100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:10:26.013085 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:10:26.013152 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:10:26.013361 (+ 209us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0621 05:10:26.013389 (+ 28us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0621 05:10:26.013480 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0621 05:10:26.013573 (+ 93us) server_negotiation.cc:299] Negotiation successful
0621 05:10:26.013598 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260621 05:10:26.013705 5885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:10:26.010739 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.131.130:35573 (local address 127.3.131.129:54613)
0621 05:10:26.010918 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0621 05:10:26.010929 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0621 05:10:26.010994 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:10:26.011125 (+ 131us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:10:26.011126 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:10:26.011127 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0621 05:10:26.011216 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:10:26.011218 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:10:26.011856 (+ 638us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:10:26.011857 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:10:26.012752 (+ 895us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:10:26.012755 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:10:26.013169 (+ 414us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:10:26.013170 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:10:26.013241 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:10:26.013256 (+ 15us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0621 05:10:26.013508 (+ 252us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0621 05:10:26.013511 (+ 3us) client_negotiation.cc:769] Sending connection context
0621 05:10:26.013549 (+ 38us) client_negotiation.cc:240] Negotiation successful
0621 05:10:26.013589 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":93,"threads_started":1}
I20260621 05:10:26.016844 5636 raft_consensus.cc:1261] T a49392eeeab6454eb7baee169e7f3a28 P 71d95c905a8f4ff181c1de9655d4a593 [term 1 FOLLOWER]: Refusing update from remote peer 1f4e2cd7408443c0a14c51ebb038ec73: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 05:10:26.016837 5776 raft_consensus.cc:1261] T a49392eeeab6454eb7baee169e7f3a28 P 8571114a663448e99821d16f67d38bcb [term 1 FOLLOWER]: Refusing update from remote peer 1f4e2cd7408443c0a14c51ebb038ec73: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 05:10:26.017185 5855 consensus_queue.cc:1035] T a49392eeeab6454eb7baee169e7f3a28 P 1f4e2cd7408443c0a14c51ebb038ec73 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8571114a663448e99821d16f67d38bcb" member_type: VOTER last_known_addr { host: "127.3.131.131" port: 43851 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 05:10:26.017302 5852 consensus_queue.cc:1035] T a49392eeeab6454eb7baee169e7f3a28 P 1f4e2cd7408443c0a14c51ebb038ec73 [LEADER]: Connected to new peer: Peer: permanent_uuid: "71d95c905a8f4ff181c1de9655d4a593" member_type: VOTER last_known_addr { host: "127.3.131.130" port: 35573 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 05:10:26.027141 5844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:10:26.024023 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.131.130:57159 (local address 127.3.131.129:45955)
0621 05:10:26.024065 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0621 05:10:26.024068 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0621 05:10:26.024374 (+ 306us) server_negotiation.cc:407] Connection header received
0621 05:10:26.024502 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:10:26.024503 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:10:26.024512 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:10:26.024554 (+ 42us) server_negotiation.cc:226] Negotiated authn=TOKEN
0621 05:10:26.024679 (+ 125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:10:26.025154 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:10:26.026308 (+ 1154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:10:26.026627 (+ 319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:10:26.026673 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:10:26.026904 (+ 231us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0621 05:10:26.026935 (+ 31us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0621 05:10:26.026980 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0621 05:10:26.027054 (+ 74us) server_negotiation.cc:299] Negotiation successful
0621 05:10:26.027074 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260621 05:10:26.027351 5890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:10:26.023981 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.131.129:45955 (local address 127.3.131.130:57159)
0621 05:10:26.024305 (+ 324us) negotiation.cc:107] Waiting for socket to connect
0621 05:10:26.024318 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0621 05:10:26.024439 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:10:26.024559 (+ 120us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:10:26.024560 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:10:26.024561 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0621 05:10:26.024616 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:10:26.024618 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:10:26.025239 (+ 621us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:10:26.025240 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:10:26.026201 (+ 961us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:10:26.026205 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:10:26.026734 (+ 529us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:10:26.026734 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:10:26.026803 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:10:26.026817 (+ 14us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0621 05:10:26.027000 (+ 183us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0621 05:10:26.027001 (+ 1us) client_negotiation.cc:769] Sending connection context
0621 05:10:26.027053 (+ 52us) client_negotiation.cc:240] Negotiation successful
0621 05:10:26.027110 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":284,"thread_start_us":86,"threads_started":1}
I20260621 05:10:26.039770 5338 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39492:
table { table_name: "test-table" } modify_external_catalogs: true
I20260621 05:10:26.039884 5338 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39492:
table { table_name: "test-table" } modify_external_catalogs: true
I20260621 05:10:26.040766 5338 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b7017628c83e4962b0fb4f3348657105: Sending DeleteTablet for 3 replicas of tablet a49392eeeab6454eb7baee169e7f3a28
I20260621 05:10:26.041102 5476 tablet_service.cc:1514] Processing DeleteTablet for tablet a49392eeeab6454eb7baee169e7f3a28 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 05:10:26 UTC) from {username='kudu', principal='kudu/127.3.131.190@KRBTEST.COM'} at 127.0.0.1:47102
I20260621 05:10:26.041433 5756 tablet_service.cc:1514] Processing DeleteTablet for tablet a49392eeeab6454eb7baee169e7f3a28 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 05:10:26 UTC) from {username='kudu', principal='kudu/127.3.131.190@KRBTEST.COM'} at 127.0.0.1:51708
I20260621 05:10:26.041456 5616 tablet_service.cc:1514] Processing DeleteTablet for tablet a49392eeeab6454eb7baee169e7f3a28 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 05:10:26 UTC) from {username='kudu', principal='kudu/127.3.131.190@KRBTEST.COM'} at 127.0.0.1:48470
I20260621 05:10:26.041708 5895 tablet_replica.cc:330] T a49392eeeab6454eb7baee169e7f3a28 P 71d95c905a8f4ff181c1de9655d4a593: stopping tablet replica
I20260621 05:10:26.041833 5895 raft_consensus.cc:2229] T a49392eeeab6454eb7baee169e7f3a28 P 71d95c905a8f4ff181c1de9655d4a593 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260621 05:10:26.041904 5895 raft_consensus.cc:2258] T a49392eeeab6454eb7baee169e7f3a28 P 71d95c905a8f4ff181c1de9655d4a593 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 05:10:26.042064 5894 tablet_replica.cc:330] T a49392eeeab6454eb7baee169e7f3a28 P 8571114a663448e99821d16f67d38bcb: stopping tablet replica
I20260621 05:10:26.042199 5894 raft_consensus.cc:2229] T a49392eeeab6454eb7baee169e7f3a28 P 8571114a663448e99821d16f67d38bcb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260621 05:10:26.042275 5894 raft_consensus.cc:2258] T a49392eeeab6454eb7baee169e7f3a28 P 8571114a663448e99821d16f67d38bcb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 05:10:26.042245 5338 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39492:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260621 05:10:26.042435 5338 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260621 05:10:26.043288 5895 ts_tablet_manager.cc:1905] T a49392eeeab6454eb7baee169e7f3a28 P 71d95c905a8f4ff181c1de9655d4a593: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 05:10:26.044025 5476 tablet_service.cc:1467] Processing CreateTablet for tablet 694d33c17de044d18e8bca5b0ca581e3 (DEFAULT_TABLE table=test-table [id=46ceaf3336ae4f64b3e83cdc4192a0bd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 05:10:26.044162 5476 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 694d33c17de044d18e8bca5b0ca581e3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 05:10:26.045437 5895 ts_tablet_manager.cc:1918] T a49392eeeab6454eb7baee169e7f3a28 P 71d95c905a8f4ff181c1de9655d4a593: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260621 05:10:26.045511 5895 log.cc:1196] T a49392eeeab6454eb7baee169e7f3a28 P 71d95c905a8f4ff181c1de9655d4a593: Deleting WAL directory at /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782018616648578-3598-0/minicluster-data/ts-1/wal/wals/a49392eeeab6454eb7baee169e7f3a28
I20260621 05:10:26.045710 5845 tablet_bootstrap.cc:492] T 694d33c17de044d18e8bca5b0ca581e3 P 1f4e2cd7408443c0a14c51ebb038ec73: Bootstrap starting.
I20260621 05:10:26.045826 5895 ts_tablet_manager.cc:1939] T a49392eeeab6454eb7baee169e7f3a28 P 71d95c905a8f4ff181c1de9655d4a593: Deleting consensus metadata
I20260621 05:10:26.046173 5845 tablet_bootstrap.cc:654] T 694d33c17de044d18e8bca5b0ca581e3 P 1f4e2cd7408443c0a14c51ebb038ec73: Neither blocks nor log segments found. Creating new log.
I20260621 05:10:26.046438 5893 tablet_replica.cc:330] T a49392eeeab6454eb7baee169e7f3a28 P 1f4e2cd7408443c0a14c51ebb038ec73: stopping tablet replica
I20260621 05:10:26.046558 5893 raft_consensus.cc:2229] T a49392eeeab6454eb7baee169e7f3a28 P 1f4e2cd7408443c0a14c51ebb038ec73 [term 1 LEADER]: Raft consensus shutting down.
I20260621 05:10:26.046677 5893 raft_consensus.cc:2258] T a49392eeeab6454eb7baee169e7f3a28 P 1f4e2cd7408443c0a14c51ebb038ec73 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 05:10:26.046851 5324 catalog_manager.cc:4841] TS 71d95c905a8f4ff181c1de9655d4a593 (127.3.131.130:35573): tablet a49392eeeab6454eb7baee169e7f3a28 (table test-table [id=e7d49a9b4b9c4111a192f51649fb628f]) successfully deleted
I20260621 05:10:26.047068 5616 tablet_service.cc:1467] Processing CreateTablet for tablet 694d33c17de044d18e8bca5b0ca581e3 (DEFAULT_TABLE table=test-table [id=46ceaf3336ae4f64b3e83cdc4192a0bd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 05:10:26.047230 5616 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 694d33c17de044d18e8bca5b0ca581e3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 05:10:26.047214 5756 tablet_service.cc:1467] Processing CreateTablet for tablet 694d33c17de044d18e8bca5b0ca581e3 (DEFAULT_TABLE table=test-table [id=46ceaf3336ae4f64b3e83cdc4192a0bd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 05:10:26.047331 5756 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 694d33c17de044d18e8bca5b0ca581e3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 05:10:26.048614 5846 tablet_bootstrap.cc:492] T 694d33c17de044d18e8bca5b0ca581e3 P 71d95c905a8f4ff181c1de9655d4a593: Bootstrap starting.
I20260621 05:10:26.048718 5847 tablet_bootstrap.cc:492] T 694d33c17de044d18e8bca5b0ca581e3 P 8571114a663448e99821d16f67d38bcb: Bootstrap starting.
I20260621 05:10:26.049196 5847 tablet_bootstrap.cc:654] T 694d33c17de044d18e8bca5b0ca581e3 P 8571114a663448e99821d16f67d38bcb: Neither blocks nor log segments found. Creating new log.
I20260621 05:10:26.049239 5846 tablet_bootstrap.cc:654] T 694d33c17de044d18e8bca5b0ca581e3 P 71d95c905a8f4ff181c1de9655d4a593: Neither blocks nor log segments found. Creating new log.
I20260621 05:10:26.049996 5893 ts_tablet_manager.cc:1905] T a49392eeeab6454eb7baee169e7f3a28 P 1f4e2cd7408443c0a14c51ebb038ec73: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 05:10:26.050974 5894 ts_tablet_manager.cc:1905] T a49392eeeab6454eb7baee169e7f3a28 P 8571114a663448e99821d16f67d38bcb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 05:10:26.051998 5893 ts_tablet_manager.cc:1918] T a49392eeeab6454eb7baee169e7f3a28 P 1f4e2cd7408443c0a14c51ebb038ec73: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260621 05:10:26.052086 5893 log.cc:1196] T a49392eeeab6454eb7baee169e7f3a28 P 1f4e2cd7408443c0a14c51ebb038ec73: Deleting WAL directory at /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782018616648578-3598-0/minicluster-data/ts-0/wal/wals/a49392eeeab6454eb7baee169e7f3a28
I20260621 05:10:26.052403 5893 ts_tablet_manager.cc:1939] T a49392eeeab6454eb7baee169e7f3a28 P 1f4e2cd7408443c0a14c51ebb038ec73: Deleting consensus metadata
I20260621 05:10:26.052881 5325 catalog_manager.cc:4841] TS 1f4e2cd7408443c0a14c51ebb038ec73 (127.3.131.129:45955): tablet a49392eeeab6454eb7baee169e7f3a28 (table test-table [id=e7d49a9b4b9c4111a192f51649fb628f]) successfully deleted
I20260621 05:10:26.053460 5894 ts_tablet_manager.cc:1918] T a49392eeeab6454eb7baee169e7f3a28 P 8571114a663448e99821d16f67d38bcb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260621 05:10:26.053565 5894 log.cc:1196] T a49392eeeab6454eb7baee169e7f3a28 P 8571114a663448e99821d16f67d38bcb: Deleting WAL directory at /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782018616648578-3598-0/minicluster-data/ts-2/wal/wals/a49392eeeab6454eb7baee169e7f3a28
I20260621 05:10:26.053843 5894 ts_tablet_manager.cc:1939] T a49392eeeab6454eb7baee169e7f3a28 P 8571114a663448e99821d16f67d38bcb: Deleting consensus metadata
I20260621 05:10:26.055042 5325 catalog_manager.cc:4841] TS 8571114a663448e99821d16f67d38bcb (127.3.131.131:43851): tablet a49392eeeab6454eb7baee169e7f3a28 (table test-table [id=e7d49a9b4b9c4111a192f51649fb628f]) successfully deleted
I20260621 05:10:26.055619 5846 tablet_bootstrap.cc:492] T 694d33c17de044d18e8bca5b0ca581e3 P 71d95c905a8f4ff181c1de9655d4a593: No bootstrap required, opened a new log
I20260621 05:10:26.055617 5847 tablet_bootstrap.cc:492] T 694d33c17de044d18e8bca5b0ca581e3 P 8571114a663448e99821d16f67d38bcb: No bootstrap required, opened a new log
I20260621 05:10:26.055617 5845 tablet_bootstrap.cc:492] T 694d33c17de044d18e8bca5b0ca581e3 P 1f4e2cd7408443c0a14c51ebb038ec73: No bootstrap required, opened a new log
I20260621 05:10:26.055694 5845 ts_tablet_manager.cc:1397] T 694d33c17de044d18e8bca5b0ca581e3 P 1f4e2cd7408443c0a14c51ebb038ec73: Time spent bootstrapping tablet: real 0.010s user 0.001s sys 0.000s
I20260621 05:10:26.055694 5847 ts_tablet_manager.cc:1397] T 694d33c17de044d18e8bca5b0ca581e3 P 8571114a663448e99821d16f67d38bcb: Time spent bootstrapping tablet: real 0.007s user 0.001s sys 0.000s
I20260621 05:10:26.055701 5846 ts_tablet_manager.cc:1397] T 694d33c17de044d18e8bca5b0ca581e3 P 71d95c905a8f4ff181c1de9655d4a593: Time spent bootstrapping tablet: real 0.007s user 0.001s sys 0.000s
I20260621 05:10:26.055881 5847 raft_consensus.cc:348] T 694d33c17de044d18e8bca5b0ca581e3 P 8571114a663448e99821d16f67d38bcb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f4e2cd7408443c0a14c51ebb038ec73" member_type: VOTER last_known_addr { host: "127.3.131.129" port: 45955 } } peers { permanent_uuid: "71d95c905a8f4ff181c1de9655d4a593" member_type: VOTER last_known_addr { host: "127.3.131.130" port: 35573 } } peers { permanent_uuid: "8571114a663448e99821d16f67d38bcb" member_type: VOTER last_known_addr { host: "127.3.131.131" port: 43851 } }
I20260621 05:10:26.055881 5845 raft_consensus.cc:348] T 694d33c17de044d18e8bca5b0ca581e3 P 1f4e2cd7408443c0a14c51ebb038ec73 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f4e2cd7408443c0a14c51ebb038ec73" member_type: VOTER last_known_addr { host: "127.3.131.129" port: 45955 } } peers { permanent_uuid: "71d95c905a8f4ff181c1de9655d4a593" member_type: VOTER last_known_addr { host: "127.3.131.130" port: 35573 } } peers { permanent_uuid: "8571114a663448e99821d16f67d38bcb" member_type: VOTER last_known_addr { host: "127.3.131.131" port: 43851 } }
I20260621 05:10:26.055963 5847 raft_consensus.cc:374] T 694d33c17de044d18e8bca5b0ca581e3 P 8571114a663448e99821d16f67d38bcb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 05:10:26.055963 5845 raft_consensus.cc:374] T 694d33c17de044d18e8bca5b0ca581e3 P 1f4e2cd7408443c0a14c51ebb038ec73 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 05:10:26.055896 5846 raft_consensus.cc:348] T 694d33c17de044d18e8bca5b0ca581e3 P 71d95c905a8f4ff181c1de9655d4a593 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f4e2cd7408443c0a14c51ebb038ec73" member_type: VOTER last_known_addr { host: "127.3.131.129" port: 45955 } } peers { permanent_uuid: "71d95c905a8f4ff181c1de9655d4a593" member_type: VOTER last_known_addr { host: "127.3.131.130" port: 35573 } } peers { permanent_uuid: "8571114a663448e99821d16f67d38bcb" member_type: VOTER last_known_addr { host: "127.3.131.131" port: 43851 } }
I20260621 05:10:26.055987 5845 raft_consensus.cc:729] T 694d33c17de044d18e8bca5b0ca581e3 P 1f4e2cd7408443c0a14c51ebb038ec73 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f4e2cd7408443c0a14c51ebb038ec73, State: Initialized, Role: FOLLOWER
I20260621 05:10:26.055987 5847 raft_consensus.cc:729] T 694d33c17de044d18e8bca5b0ca581e3 P 8571114a663448e99821d16f67d38bcb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8571114a663448e99821d16f67d38bcb, State: Initialized, Role: FOLLOWER
I20260621 05:10:26.055998 5846 raft_consensus.cc:374] T 694d33c17de044d18e8bca5b0ca581e3 P 71d95c905a8f4ff181c1de9655d4a593 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 05:10:26.056034 5846 raft_consensus.cc:729] T 694d33c17de044d18e8bca5b0ca581e3 P 71d95c905a8f4ff181c1de9655d4a593 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 71d95c905a8f4ff181c1de9655d4a593, State: Initialized, Role: FOLLOWER
I20260621 05:10:26.056031 5847 consensus_queue.cc:260] T 694d33c17de044d18e8bca5b0ca581e3 P 8571114a663448e99821d16f67d38bcb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f4e2cd7408443c0a14c51ebb038ec73" member_type: VOTER last_known_addr { host: "127.3.131.129" port: 45955 } } peers { permanent_uuid: "71d95c905a8f4ff181c1de9655d4a593" member_type: VOTER last_known_addr { host: "127.3.131.130" port: 35573 } } peers { permanent_uuid: "8571114a663448e99821d16f67d38bcb" member_type: VOTER last_known_addr { host: "127.3.131.131" port: 43851 } }
I20260621 05:10:26.056031 5845 consensus_queue.cc:260] T 694d33c17de044d18e8bca5b0ca581e3 P 1f4e2cd7408443c0a14c51ebb038ec73 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f4e2cd7408443c0a14c51ebb038ec73" member_type: VOTER last_known_addr { host: "127.3.131.129" port: 45955 } } peers { permanent_uuid: "71d95c905a8f4ff181c1de9655d4a593" member_type: VOTER last_known_addr { host: "127.3.131.130" port: 35573 } } peers { permanent_uuid: "8571114a663448e99821d16f67d38bcb" member_type: VOTER last_known_addr { host: "127.3.131.131" port: 43851 } }
I20260621 05:10:26.056082 5846 consensus_queue.cc:260] T 694d33c17de044d18e8bca5b0ca581e3 P 71d95c905a8f4ff181c1de9655d4a593 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f4e2cd7408443c0a14c51ebb038ec73" member_type: VOTER last_known_addr { host: "127.3.131.129" port: 45955 } } peers { permanent_uuid: "71d95c905a8f4ff181c1de9655d4a593" member_type: VOTER last_known_addr { host: "127.3.131.130" port: 35573 } } peers { permanent_uuid: "8571114a663448e99821d16f67d38bcb" member_type: VOTER last_known_addr { host: "127.3.131.131" port: 43851 } }
I20260621 05:10:26.056126 5845 ts_tablet_manager.cc:1428] T 694d33c17de044d18e8bca5b0ca581e3 P 1f4e2cd7408443c0a14c51ebb038ec73: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260621 05:10:26.056133 5847 ts_tablet_manager.cc:1428] T 694d33c17de044d18e8bca5b0ca581e3 P 8571114a663448e99821d16f67d38bcb: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260621 05:10:26.056206 5846 ts_tablet_manager.cc:1428] T 694d33c17de044d18e8bca5b0ca581e3 P 71d95c905a8f4ff181c1de9655d4a593: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260621 05:10:26.174777 5888 raft_consensus.cc:482] T 694d33c17de044d18e8bca5b0ca581e3 P 1f4e2cd7408443c0a14c51ebb038ec73 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260621 05:10:26.174897 5888 raft_consensus.cc:504] T 694d33c17de044d18e8bca5b0ca581e3 P 1f4e2cd7408443c0a14c51ebb038ec73 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f4e2cd7408443c0a14c51ebb038ec73" member_type: VOTER last_known_addr { host: "127.3.131.129" port: 45955 } } peers { permanent_uuid: "71d95c905a8f4ff181c1de9655d4a593" member_type: VOTER last_known_addr { host: "127.3.131.130" port: 35573 } } peers { permanent_uuid: "8571114a663448e99821d16f67d38bcb" member_type: VOTER last_known_addr { host: "127.3.131.131" port: 43851 } }
I20260621 05:10:26.175061 5888 leader_election.cc:288] T 694d33c17de044d18e8bca5b0ca581e3 P 1f4e2cd7408443c0a14c51ebb038ec73 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 71d95c905a8f4ff181c1de9655d4a593 (127.3.131.130:35573), 8571114a663448e99821d16f67d38bcb (127.3.131.131:43851)
I20260621 05:10:26.175312 5776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "694d33c17de044d18e8bca5b0ca581e3" candidate_uuid: "1f4e2cd7408443c0a14c51ebb038ec73" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8571114a663448e99821d16f67d38bcb" is_pre_election: true
I20260621 05:10:26.175312 5636 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "694d33c17de044d18e8bca5b0ca581e3" candidate_uuid: "1f4e2cd7408443c0a14c51ebb038ec73" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "71d95c905a8f4ff181c1de9655d4a593" is_pre_election: true
I20260621 05:10:26.175421 5776 raft_consensus.cc:2454] T 694d33c17de044d18e8bca5b0ca581e3 P 8571114a663448e99821d16f67d38bcb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1f4e2cd7408443c0a14c51ebb038ec73 in term 0.
I20260621 05:10:26.175421 5636 raft_consensus.cc:2454] T 694d33c17de044d18e8bca5b0ca581e3 P 71d95c905a8f4ff181c1de9655d4a593 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1f4e2cd7408443c0a14c51ebb038ec73 in term 0.
I20260621 05:10:26.175582 5428 leader_election.cc:302] T 694d33c17de044d18e8bca5b0ca581e3 P 1f4e2cd7408443c0a14c51ebb038ec73 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f4e2cd7408443c0a14c51ebb038ec73, 71d95c905a8f4ff181c1de9655d4a593; no voters:
I20260621 05:10:26.175684 5888 raft_consensus.cc:2781] T 694d33c17de044d18e8bca5b0ca581e3 P 1f4e2cd7408443c0a14c51ebb038ec73 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260621 05:10:26.175729 5888 raft_consensus.cc:482] T 694d33c17de044d18e8bca5b0ca581e3 P 1f4e2cd7408443c0a14c51ebb038ec73 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260621 05:10:26.175762 5888 raft_consensus.cc:3037] T 694d33c17de044d18e8bca5b0ca581e3 P 1f4e2cd7408443c0a14c51ebb038ec73 [term 0 FOLLOWER]: Advancing to term 1
I20260621 05:10:26.176481 5888 raft_consensus.cc:504] T 694d33c17de044d18e8bca5b0ca581e3 P 1f4e2cd7408443c0a14c51ebb038ec73 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f4e2cd7408443c0a14c51ebb038ec73" member_type: VOTER last_known_addr { host: "127.3.131.129" port: 45955 } } peers { permanent_uuid: "71d95c905a8f4ff181c1de9655d4a593" member_type: VOTER last_known_addr { host: "127.3.131.130" port: 35573 } } peers { permanent_uuid: "8571114a663448e99821d16f67d38bcb" member_type: VOTER last_known_addr { host: "127.3.131.131" port: 43851 } }
I20260621 05:10:26.176631 5888 leader_election.cc:288] T 694d33c17de044d18e8bca5b0ca581e3 P 1f4e2cd7408443c0a14c51ebb038ec73 [CANDIDATE]: Term 1 election: Requested vote from peers 71d95c905a8f4ff181c1de9655d4a593 (127.3.131.130:35573), 8571114a663448e99821d16f67d38bcb (127.3.131.131:43851)
I20260621 05:10:26.176761 5636 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "694d33c17de044d18e8bca5b0ca581e3" candidate_uuid: "1f4e2cd7408443c0a14c51ebb038ec73" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "71d95c905a8f4ff181c1de9655d4a593"
I20260621 05:10:26.176846 5636 raft_consensus.cc:3037] T 694d33c17de044d18e8bca5b0ca581e3 P 71d95c905a8f4ff181c1de9655d4a593 [term 0 FOLLOWER]: Advancing to term 1
I20260621 05:10:26.176815 5776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "694d33c17de044d18e8bca5b0ca581e3" candidate_uuid: "1f4e2cd7408443c0a14c51ebb038ec73" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8571114a663448e99821d16f67d38bcb"
I20260621 05:10:26.176877 5776 raft_consensus.cc:3037] T 694d33c17de044d18e8bca5b0ca581e3 P 8571114a663448e99821d16f67d38bcb [term 0 FOLLOWER]: Advancing to term 1
I20260621 05:10:26.177626 5636 raft_consensus.cc:2454] T 694d33c17de044d18e8bca5b0ca581e3 P 71d95c905a8f4ff181c1de9655d4a593 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1f4e2cd7408443c0a14c51ebb038ec73 in term 1.
I20260621 05:10:26.177644 5776 raft_consensus.cc:2454] T 694d33c17de044d18e8bca5b0ca581e3 P 8571114a663448e99821d16f67d38bcb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1f4e2cd7408443c0a14c51ebb038ec73 in term 1.
I20260621 05:10:26.177789 5429 leader_election.cc:302] T 694d33c17de044d18e8bca5b0ca581e3 P 1f4e2cd7408443c0a14c51ebb038ec73 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f4e2cd7408443c0a14c51ebb038ec73, 8571114a663448e99821d16f67d38bcb; no voters:
I20260621 05:10:26.177917 5888 raft_consensus.cc:2781] T 694d33c17de044d18e8bca5b0ca581e3 P 1f4e2cd7408443c0a14c51ebb038ec73 [term 1 FOLLOWER]: Leader election won for term 1
I20260621 05:10:26.177978 5888 raft_consensus.cc:686] T 694d33c17de044d18e8bca5b0ca581e3 P 1f4e2cd7408443c0a14c51ebb038ec73 [term 1 LEADER]: Becoming Leader. State: Replica: 1f4e2cd7408443c0a14c51ebb038ec73, State: Running, Role: LEADER
I20260621 05:10:26.178027 5888 consensus_queue.cc:237] T 694d33c17de044d18e8bca5b0ca581e3 P 1f4e2cd7408443c0a14c51ebb038ec73 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f4e2cd7408443c0a14c51ebb038ec73" member_type: VOTER last_known_addr { host: "127.3.131.129" port: 45955 } } peers { permanent_uuid: "71d95c905a8f4ff181c1de9655d4a593" member_type: VOTER last_known_addr { host: "127.3.131.130" port: 35573 } } peers { permanent_uuid: "8571114a663448e99821d16f67d38bcb" member_type: VOTER last_known_addr { host: "127.3.131.131" port: 43851 } }
I20260621 05:10:26.178714 5338 catalog_manager.cc:5495] T 694d33c17de044d18e8bca5b0ca581e3 P 1f4e2cd7408443c0a14c51ebb038ec73 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1f4e2cd7408443c0a14c51ebb038ec73 (127.3.131.129). New cstate: current_term: 1 leader_uuid: "1f4e2cd7408443c0a14c51ebb038ec73" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f4e2cd7408443c0a14c51ebb038ec73" member_type: VOTER last_known_addr { host: "127.3.131.129" port: 45955 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "71d95c905a8f4ff181c1de9655d4a593" member_type: VOTER last_known_addr { host: "127.3.131.130" port: 35573 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8571114a663448e99821d16f67d38bcb" member_type: VOTER last_known_addr { host: "127.3.131.131" port: 43851 } health_report { overall_health: UNKNOWN } } }
I20260621 05:10:26.203253 5776 raft_consensus.cc:1261] T 694d33c17de044d18e8bca5b0ca581e3 P 8571114a663448e99821d16f67d38bcb [term 1 FOLLOWER]: Refusing update from remote peer 1f4e2cd7408443c0a14c51ebb038ec73: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 05:10:26.203254 5636 raft_consensus.cc:1261] T 694d33c17de044d18e8bca5b0ca581e3 P 71d95c905a8f4ff181c1de9655d4a593 [term 1 FOLLOWER]: Refusing update from remote peer 1f4e2cd7408443c0a14c51ebb038ec73: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 05:10:26.203500 5888 consensus_queue.cc:1035] T 694d33c17de044d18e8bca5b0ca581e3 P 1f4e2cd7408443c0a14c51ebb038ec73 [LEADER]: Connected to new peer: Peer: permanent_uuid: "71d95c905a8f4ff181c1de9655d4a593" member_type: VOTER last_known_addr { host: "127.3.131.130" port: 35573 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 05:10:26.203636 5855 consensus_queue.cc:1035] T 694d33c17de044d18e8bca5b0ca581e3 P 1f4e2cd7408443c0a14c51ebb038ec73 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8571114a663448e99821d16f67d38bcb" member_type: VOTER last_known_addr { host: "127.3.131.131" port: 43851 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 05:10:26.205367 5876 mvcc.cc:205] Tried to move back new op lower bound from 7299148292926812160 to 7299148292825473024. Current Snapshot: MvccSnapshot[applied={T|T < 7299148292926812160}]
W20260621 05:10:26.209372 5874 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260621 05:10:27.729825 5337 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39492:
table { table_name: "test-table" } modify_external_catalogs: true
I20260621 05:10:27.729957 5337 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39492:
table { table_name: "test-table" } modify_external_catalogs: true
I20260621 05:10:27.730887 5337 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b7017628c83e4962b0fb4f3348657105: Sending DeleteTablet for 3 replicas of tablet 694d33c17de044d18e8bca5b0ca581e3
I20260621 05:10:27.731281 5476 tablet_service.cc:1514] Processing DeleteTablet for tablet 694d33c17de044d18e8bca5b0ca581e3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 05:10:27 UTC) from {username='kudu', principal='kudu/127.3.131.190@KRBTEST.COM'} at 127.0.0.1:47102
I20260621 05:10:27.731302 5616 tablet_service.cc:1514] Processing DeleteTablet for tablet 694d33c17de044d18e8bca5b0ca581e3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 05:10:27 UTC) from {username='kudu', principal='kudu/127.3.131.190@KRBTEST.COM'} at 127.0.0.1:48470
I20260621 05:10:27.731485 5756 tablet_service.cc:1514] Processing DeleteTablet for tablet 694d33c17de044d18e8bca5b0ca581e3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 05:10:27 UTC) from {username='kudu', principal='kudu/127.3.131.190@KRBTEST.COM'} at 127.0.0.1:51708
I20260621 05:10:27.731701 5925 tablet_replica.cc:330] T 694d33c17de044d18e8bca5b0ca581e3 P 8571114a663448e99821d16f67d38bcb: stopping tablet replica
I20260621 05:10:27.731765 5925 raft_consensus.cc:2229] T 694d33c17de044d18e8bca5b0ca581e3 P 8571114a663448e99821d16f67d38bcb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260621 05:10:27.731807 5925 raft_consensus.cc:2258] T 694d33c17de044d18e8bca5b0ca581e3 P 8571114a663448e99821d16f67d38bcb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 05:10:27.732060 5925 ts_tablet_manager.cc:1905] T 694d33c17de044d18e8bca5b0ca581e3 P 8571114a663448e99821d16f67d38bcb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 05:10:27.733507 5925 ts_tablet_manager.cc:1918] T 694d33c17de044d18e8bca5b0ca581e3 P 8571114a663448e99821d16f67d38bcb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260621 05:10:27.733549 5925 log.cc:1196] T 694d33c17de044d18e8bca5b0ca581e3 P 8571114a663448e99821d16f67d38bcb: Deleting WAL directory at /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782018616648578-3598-0/minicluster-data/ts-2/wal/wals/694d33c17de044d18e8bca5b0ca581e3
I20260621 05:10:27.733726 5925 ts_tablet_manager.cc:1939] T 694d33c17de044d18e8bca5b0ca581e3 P 8571114a663448e99821d16f67d38bcb: Deleting consensus metadata
I20260621 05:10:27.733821 5923 tablet_replica.cc:330] T 694d33c17de044d18e8bca5b0ca581e3 P 1f4e2cd7408443c0a14c51ebb038ec73: stopping tablet replica
I20260621 05:10:27.733899 5923 raft_consensus.cc:2229] T 694d33c17de044d18e8bca5b0ca581e3 P 1f4e2cd7408443c0a14c51ebb038ec73 [term 1 LEADER]: Raft consensus shutting down.
I20260621 05:10:27.733973 5923 raft_consensus.cc:2258] T 694d33c17de044d18e8bca5b0ca581e3 P 1f4e2cd7408443c0a14c51ebb038ec73 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 05:10:27.734061 5325 catalog_manager.cc:4841] TS 8571114a663448e99821d16f67d38bcb (127.3.131.131:43851): tablet 694d33c17de044d18e8bca5b0ca581e3 (table test-table [id=46ceaf3336ae4f64b3e83cdc4192a0bd]) successfully deleted
I20260621 05:10:27.734292 5923 ts_tablet_manager.cc:1905] T 694d33c17de044d18e8bca5b0ca581e3 P 1f4e2cd7408443c0a14c51ebb038ec73: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 05:10:27.735339 5924 tablet_replica.cc:330] T 694d33c17de044d18e8bca5b0ca581e3 P 71d95c905a8f4ff181c1de9655d4a593: stopping tablet replica
I20260621 05:10:27.735428 5924 raft_consensus.cc:2229] T 694d33c17de044d18e8bca5b0ca581e3 P 71d95c905a8f4ff181c1de9655d4a593 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260621 05:10:27.735481 5924 raft_consensus.cc:2258] T 694d33c17de044d18e8bca5b0ca581e3 P 71d95c905a8f4ff181c1de9655d4a593 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 05:10:27.735766 5924 ts_tablet_manager.cc:1905] T 694d33c17de044d18e8bca5b0ca581e3 P 71d95c905a8f4ff181c1de9655d4a593: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 05:10:27.736275 5923 ts_tablet_manager.cc:1918] T 694d33c17de044d18e8bca5b0ca581e3 P 1f4e2cd7408443c0a14c51ebb038ec73: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260621 05:10:27.736341 5923 log.cc:1196] T 694d33c17de044d18e8bca5b0ca581e3 P 1f4e2cd7408443c0a14c51ebb038ec73: Deleting WAL directory at /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782018616648578-3598-0/minicluster-data/ts-0/wal/wals/694d33c17de044d18e8bca5b0ca581e3
I20260621 05:10:27.736589 5923 ts_tablet_manager.cc:1939] T 694d33c17de044d18e8bca5b0ca581e3 P 1f4e2cd7408443c0a14c51ebb038ec73: Deleting consensus metadata
I20260621 05:10:27.736917 5325 catalog_manager.cc:4841] TS 1f4e2cd7408443c0a14c51ebb038ec73 (127.3.131.129:45955): tablet 694d33c17de044d18e8bca5b0ca581e3 (table test-table [id=46ceaf3336ae4f64b3e83cdc4192a0bd]) successfully deleted
I20260621 05:10:27.737429 5924 ts_tablet_manager.cc:1918] T 694d33c17de044d18e8bca5b0ca581e3 P 71d95c905a8f4ff181c1de9655d4a593: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260621 05:10:27.737478 5924 log.cc:1196] T 694d33c17de044d18e8bca5b0ca581e3 P 71d95c905a8f4ff181c1de9655d4a593: Deleting WAL directory at /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782018616648578-3598-0/minicluster-data/ts-1/wal/wals/694d33c17de044d18e8bca5b0ca581e3
I20260621 05:10:27.737650 5924 ts_tablet_manager.cc:1939] T 694d33c17de044d18e8bca5b0ca581e3 P 71d95c905a8f4ff181c1de9655d4a593: Deleting consensus metadata
I20260621 05:10:27.738155 5324 catalog_manager.cc:4841] TS 71d95c905a8f4ff181c1de9655d4a593 (127.3.131.130:35573): tablet 694d33c17de044d18e8bca5b0ca581e3 (table test-table [id=46ceaf3336ae4f64b3e83cdc4192a0bd]) successfully deleted
Jun 21 05:10:27 dist-test-slave-vrq1 krb5kdc[5294](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782018627, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 21 05:10:27 dist-test-slave-vrq1 krb5kdc[5294](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782018627, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.3.131.190@KRBTEST.COM
I20260621 05:10:27.762576 5933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:10:27.749505 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39504 (local address 127.3.131.190:43783)
0621 05:10:27.749649 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0621 05:10:27.749651 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:10:27.749661 (+ 10us) server_negotiation.cc:407] Connection header received
0621 05:10:27.749727 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:10:27.749728 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:10:27.749739 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:10:27.749806 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:10:27.750353 (+ 547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:10:27.750854 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:10:27.751649 (+ 795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:10:27.751859 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:10:27.751922 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:10:27.759731 (+ 7809us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:10:27.759733 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:10:27.759733 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:10:27.759742 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:10:27.761461 (+ 1719us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:10:27.761825 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:10:27.761826 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:10:27.761826 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:10:27.761880 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:10:27.762077 (+ 197us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:10:27.762078 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:10:27.762078 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:10:27.762223 (+ 145us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:10:27.762316 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0621 05:10:27.762423 (+ 107us) server_negotiation.cc:299] Negotiation successful
0621 05:10:27.762500 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":113,"thread_start_us":89,"threads_started":1}
I20260621 05:10:27.764357 5337 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:39504:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260621 05:10:27.764513 5337 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260621 05:10:27.766062 5616 tablet_service.cc:1467] Processing CreateTablet for tablet 8f4d1fd6d2944817a641f0492c74cf07 (DEFAULT_TABLE table=test-table [id=40c1f47a698c467087d6e37260b7ecea]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 05:10:27.766093 5476 tablet_service.cc:1467] Processing CreateTablet for tablet 8f4d1fd6d2944817a641f0492c74cf07 (DEFAULT_TABLE table=test-table [id=40c1f47a698c467087d6e37260b7ecea]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 05:10:27.766213 5476 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8f4d1fd6d2944817a641f0492c74cf07. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 05:10:27.766213 5616 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8f4d1fd6d2944817a641f0492c74cf07. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 05:10:27.767339 5756 tablet_service.cc:1467] Processing CreateTablet for tablet 8f4d1fd6d2944817a641f0492c74cf07 (DEFAULT_TABLE table=test-table [id=40c1f47a698c467087d6e37260b7ecea]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 05:10:27.767479 5756 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8f4d1fd6d2944817a641f0492c74cf07. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 05:10:27.767748 5934 tablet_bootstrap.cc:492] T 8f4d1fd6d2944817a641f0492c74cf07 P 1f4e2cd7408443c0a14c51ebb038ec73: Bootstrap starting.
I20260621 05:10:27.767802 5935 tablet_bootstrap.cc:492] T 8f4d1fd6d2944817a641f0492c74cf07 P 71d95c905a8f4ff181c1de9655d4a593: Bootstrap starting.
I20260621 05:10:27.768146 5935 tablet_bootstrap.cc:654] T 8f4d1fd6d2944817a641f0492c74cf07 P 71d95c905a8f4ff181c1de9655d4a593: Neither blocks nor log segments found. Creating new log.
I20260621 05:10:27.768146 5934 tablet_bootstrap.cc:654] T 8f4d1fd6d2944817a641f0492c74cf07 P 1f4e2cd7408443c0a14c51ebb038ec73: Neither blocks nor log segments found. Creating new log.
I20260621 05:10:27.768735 5935 tablet_bootstrap.cc:492] T 8f4d1fd6d2944817a641f0492c74cf07 P 71d95c905a8f4ff181c1de9655d4a593: No bootstrap required, opened a new log
I20260621 05:10:27.768805 5935 ts_tablet_manager.cc:1397] T 8f4d1fd6d2944817a641f0492c74cf07 P 71d95c905a8f4ff181c1de9655d4a593: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260621 05:10:27.768816 5934 tablet_bootstrap.cc:492] T 8f4d1fd6d2944817a641f0492c74cf07 P 1f4e2cd7408443c0a14c51ebb038ec73: No bootstrap required, opened a new log
I20260621 05:10:27.768847 5938 tablet_bootstrap.cc:492] T 8f4d1fd6d2944817a641f0492c74cf07 P 8571114a663448e99821d16f67d38bcb: Bootstrap starting.
I20260621 05:10:27.768868 5934 ts_tablet_manager.cc:1397] T 8f4d1fd6d2944817a641f0492c74cf07 P 1f4e2cd7408443c0a14c51ebb038ec73: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260621 05:10:27.768925 5935 raft_consensus.cc:348] T 8f4d1fd6d2944817a641f0492c74cf07 P 71d95c905a8f4ff181c1de9655d4a593 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f4e2cd7408443c0a14c51ebb038ec73" member_type: VOTER last_known_addr { host: "127.3.131.129" port: 45955 } } peers { permanent_uuid: "71d95c905a8f4ff181c1de9655d4a593" member_type: VOTER last_known_addr { host: "127.3.131.130" port: 35573 } } peers { permanent_uuid: "8571114a663448e99821d16f67d38bcb" member_type: VOTER last_known_addr { host: "127.3.131.131" port: 43851 } }
I20260621 05:10:27.768975 5934 raft_consensus.cc:348] T 8f4d1fd6d2944817a641f0492c74cf07 P 1f4e2cd7408443c0a14c51ebb038ec73 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f4e2cd7408443c0a14c51ebb038ec73" member_type: VOTER last_known_addr { host: "127.3.131.129" port: 45955 } } peers { permanent_uuid: "71d95c905a8f4ff181c1de9655d4a593" member_type: VOTER last_known_addr { host: "127.3.131.130" port: 35573 } } peers { permanent_uuid: "8571114a663448e99821d16f67d38bcb" member_type: VOTER last_known_addr { host: "127.3.131.131" port: 43851 } }
I20260621 05:10:27.769003 5935 raft_consensus.cc:374] T 8f4d1fd6d2944817a641f0492c74cf07 P 71d95c905a8f4ff181c1de9655d4a593 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 05:10:27.769037 5934 raft_consensus.cc:374] T 8f4d1fd6d2944817a641f0492c74cf07 P 1f4e2cd7408443c0a14c51ebb038ec73 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 05:10:27.769050 5934 raft_consensus.cc:729] T 8f4d1fd6d2944817a641f0492c74cf07 P 1f4e2cd7408443c0a14c51ebb038ec73 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f4e2cd7408443c0a14c51ebb038ec73, State: Initialized, Role: FOLLOWER
I20260621 05:10:27.769048 5935 raft_consensus.cc:729] T 8f4d1fd6d2944817a641f0492c74cf07 P 71d95c905a8f4ff181c1de9655d4a593 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 71d95c905a8f4ff181c1de9655d4a593, State: Initialized, Role: FOLLOWER
I20260621 05:10:27.769098 5935 consensus_queue.cc:260] T 8f4d1fd6d2944817a641f0492c74cf07 P 71d95c905a8f4ff181c1de9655d4a593 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f4e2cd7408443c0a14c51ebb038ec73" member_type: VOTER last_known_addr { host: "127.3.131.129" port: 45955 } } peers { permanent_uuid: "71d95c905a8f4ff181c1de9655d4a593" member_type: VOTER last_known_addr { host: "127.3.131.130" port: 35573 } } peers { permanent_uuid: "8571114a663448e99821d16f67d38bcb" member_type: VOTER last_known_addr { host: "127.3.131.131" port: 43851 } }
I20260621 05:10:27.769112 5934 consensus_queue.cc:260] T 8f4d1fd6d2944817a641f0492c74cf07 P 1f4e2cd7408443c0a14c51ebb038ec73 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f4e2cd7408443c0a14c51ebb038ec73" member_type: VOTER last_known_addr { host: "127.3.131.129" port: 45955 } } peers { permanent_uuid: "71d95c905a8f4ff181c1de9655d4a593" member_type: VOTER last_known_addr { host: "127.3.131.130" port: 35573 } } peers { permanent_uuid: "8571114a663448e99821d16f67d38bcb" member_type: VOTER last_known_addr { host: "127.3.131.131" port: 43851 } }
I20260621 05:10:27.769198 5935 ts_tablet_manager.cc:1428] T 8f4d1fd6d2944817a641f0492c74cf07 P 71d95c905a8f4ff181c1de9655d4a593: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260621 05:10:27.769212 5934 ts_tablet_manager.cc:1428] T 8f4d1fd6d2944817a641f0492c74cf07 P 1f4e2cd7408443c0a14c51ebb038ec73: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260621 05:10:27.769248 5938 tablet_bootstrap.cc:654] T 8f4d1fd6d2944817a641f0492c74cf07 P 8571114a663448e99821d16f67d38bcb: Neither blocks nor log segments found. Creating new log.
I20260621 05:10:27.770013 5938 tablet_bootstrap.cc:492] T 8f4d1fd6d2944817a641f0492c74cf07 P 8571114a663448e99821d16f67d38bcb: No bootstrap required, opened a new log
I20260621 05:10:27.770058 5938 ts_tablet_manager.cc:1397] T 8f4d1fd6d2944817a641f0492c74cf07 P 8571114a663448e99821d16f67d38bcb: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260621 05:10:27.770228 5938 raft_consensus.cc:348] T 8f4d1fd6d2944817a641f0492c74cf07 P 8571114a663448e99821d16f67d38bcb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f4e2cd7408443c0a14c51ebb038ec73" member_type: VOTER last_known_addr { host: "127.3.131.129" port: 45955 } } peers { permanent_uuid: "71d95c905a8f4ff181c1de9655d4a593" member_type: VOTER last_known_addr { host: "127.3.131.130" port: 35573 } } peers { permanent_uuid: "8571114a663448e99821d16f67d38bcb" member_type: VOTER last_known_addr { host: "127.3.131.131" port: 43851 } }
I20260621 05:10:27.770283 5938 raft_consensus.cc:374] T 8f4d1fd6d2944817a641f0492c74cf07 P 8571114a663448e99821d16f67d38bcb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 05:10:27.770298 5938 raft_consensus.cc:729] T 8f4d1fd6d2944817a641f0492c74cf07 P 8571114a663448e99821d16f67d38bcb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8571114a663448e99821d16f67d38bcb, State: Initialized, Role: FOLLOWER
I20260621 05:10:27.770351 5938 consensus_queue.cc:260] T 8f4d1fd6d2944817a641f0492c74cf07 P 8571114a663448e99821d16f67d38bcb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f4e2cd7408443c0a14c51ebb038ec73" member_type: VOTER last_known_addr { host: "127.3.131.129" port: 45955 } } peers { permanent_uuid: "71d95c905a8f4ff181c1de9655d4a593" member_type: VOTER last_known_addr { host: "127.3.131.130" port: 35573 } } peers { permanent_uuid: "8571114a663448e99821d16f67d38bcb" member_type: VOTER last_known_addr { host: "127.3.131.131" port: 43851 } }
I20260621 05:10:27.770534 5938 ts_tablet_manager.cc:1428] T 8f4d1fd6d2944817a641f0492c74cf07 P 8571114a663448e99821d16f67d38bcb: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260621 05:10:27.898394 5868 raft_consensus.cc:482] T 8f4d1fd6d2944817a641f0492c74cf07 P 71d95c905a8f4ff181c1de9655d4a593 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260621 05:10:27.898566 5868 raft_consensus.cc:504] T 8f4d1fd6d2944817a641f0492c74cf07 P 71d95c905a8f4ff181c1de9655d4a593 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f4e2cd7408443c0a14c51ebb038ec73" member_type: VOTER last_known_addr { host: "127.3.131.129" port: 45955 } } peers { permanent_uuid: "71d95c905a8f4ff181c1de9655d4a593" member_type: VOTER last_known_addr { host: "127.3.131.130" port: 35573 } } peers { permanent_uuid: "8571114a663448e99821d16f67d38bcb" member_type: VOTER last_known_addr { host: "127.3.131.131" port: 43851 } }
I20260621 05:10:27.898809 5868 leader_election.cc:288] T 8f4d1fd6d2944817a641f0492c74cf07 P 71d95c905a8f4ff181c1de9655d4a593 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1f4e2cd7408443c0a14c51ebb038ec73 (127.3.131.129:45955), 8571114a663448e99821d16f67d38bcb (127.3.131.131:43851)
I20260621 05:10:27.899024 5496 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f4d1fd6d2944817a641f0492c74cf07" candidate_uuid: "71d95c905a8f4ff181c1de9655d4a593" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f4e2cd7408443c0a14c51ebb038ec73" is_pre_election: true
I20260621 05:10:27.899129 5496 raft_consensus.cc:2454] T 8f4d1fd6d2944817a641f0492c74cf07 P 1f4e2cd7408443c0a14c51ebb038ec73 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 71d95c905a8f4ff181c1de9655d4a593 in term 0.
I20260621 05:10:27.899142 5776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f4d1fd6d2944817a641f0492c74cf07" candidate_uuid: "71d95c905a8f4ff181c1de9655d4a593" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8571114a663448e99821d16f67d38bcb" is_pre_election: true
I20260621 05:10:27.899307 5776 raft_consensus.cc:2454] T 8f4d1fd6d2944817a641f0492c74cf07 P 8571114a663448e99821d16f67d38bcb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 71d95c905a8f4ff181c1de9655d4a593 in term 0.
I20260621 05:10:27.899366 5569 leader_election.cc:302] T 8f4d1fd6d2944817a641f0492c74cf07 P 71d95c905a8f4ff181c1de9655d4a593 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f4e2cd7408443c0a14c51ebb038ec73, 71d95c905a8f4ff181c1de9655d4a593; no voters:
I20260621 05:10:27.899500 5868 raft_consensus.cc:2781] T 8f4d1fd6d2944817a641f0492c74cf07 P 71d95c905a8f4ff181c1de9655d4a593 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260621 05:10:27.899554 5868 raft_consensus.cc:482] T 8f4d1fd6d2944817a641f0492c74cf07 P 71d95c905a8f4ff181c1de9655d4a593 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260621 05:10:27.899578 5868 raft_consensus.cc:3037] T 8f4d1fd6d2944817a641f0492c74cf07 P 71d95c905a8f4ff181c1de9655d4a593 [term 0 FOLLOWER]: Advancing to term 1
I20260621 05:10:27.900321 5868 raft_consensus.cc:504] T 8f4d1fd6d2944817a641f0492c74cf07 P 71d95c905a8f4ff181c1de9655d4a593 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f4e2cd7408443c0a14c51ebb038ec73" member_type: VOTER last_known_addr { host: "127.3.131.129" port: 45955 } } peers { permanent_uuid: "71d95c905a8f4ff181c1de9655d4a593" member_type: VOTER last_known_addr { host: "127.3.131.130" port: 35573 } } peers { permanent_uuid: "8571114a663448e99821d16f67d38bcb" member_type: VOTER last_known_addr { host: "127.3.131.131" port: 43851 } }
I20260621 05:10:27.900425 5868 leader_election.cc:288] T 8f4d1fd6d2944817a641f0492c74cf07 P 71d95c905a8f4ff181c1de9655d4a593 [CANDIDATE]: Term 1 election: Requested vote from peers 1f4e2cd7408443c0a14c51ebb038ec73 (127.3.131.129:45955), 8571114a663448e99821d16f67d38bcb (127.3.131.131:43851)
I20260621 05:10:27.900647 5496 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f4d1fd6d2944817a641f0492c74cf07" candidate_uuid: "71d95c905a8f4ff181c1de9655d4a593" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f4e2cd7408443c0a14c51ebb038ec73"
I20260621 05:10:27.900693 5776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f4d1fd6d2944817a641f0492c74cf07" candidate_uuid: "71d95c905a8f4ff181c1de9655d4a593" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8571114a663448e99821d16f67d38bcb"
I20260621 05:10:27.900736 5496 raft_consensus.cc:3037] T 8f4d1fd6d2944817a641f0492c74cf07 P 1f4e2cd7408443c0a14c51ebb038ec73 [term 0 FOLLOWER]: Advancing to term 1
I20260621 05:10:27.900761 5776 raft_consensus.cc:3037] T 8f4d1fd6d2944817a641f0492c74cf07 P 8571114a663448e99821d16f67d38bcb [term 0 FOLLOWER]: Advancing to term 1
I20260621 05:10:27.901510 5496 raft_consensus.cc:2454] T 8f4d1fd6d2944817a641f0492c74cf07 P 1f4e2cd7408443c0a14c51ebb038ec73 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 71d95c905a8f4ff181c1de9655d4a593 in term 1.
I20260621 05:10:27.901510 5776 raft_consensus.cc:2454] T 8f4d1fd6d2944817a641f0492c74cf07 P 8571114a663448e99821d16f67d38bcb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 71d95c905a8f4ff181c1de9655d4a593 in term 1.
I20260621 05:10:27.901744 5569 leader_election.cc:302] T 8f4d1fd6d2944817a641f0492c74cf07 P 71d95c905a8f4ff181c1de9655d4a593 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 71d95c905a8f4ff181c1de9655d4a593, 8571114a663448e99821d16f67d38bcb; no voters:
I20260621 05:10:27.901978 5868 raft_consensus.cc:2781] T 8f4d1fd6d2944817a641f0492c74cf07 P 71d95c905a8f4ff181c1de9655d4a593 [term 1 FOLLOWER]: Leader election won for term 1
I20260621 05:10:27.902047 5868 raft_consensus.cc:686] T 8f4d1fd6d2944817a641f0492c74cf07 P 71d95c905a8f4ff181c1de9655d4a593 [term 1 LEADER]: Becoming Leader. State: Replica: 71d95c905a8f4ff181c1de9655d4a593, State: Running, Role: LEADER
I20260621 05:10:27.902109 5868 consensus_queue.cc:237] T 8f4d1fd6d2944817a641f0492c74cf07 P 71d95c905a8f4ff181c1de9655d4a593 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f4e2cd7408443c0a14c51ebb038ec73" member_type: VOTER last_known_addr { host: "127.3.131.129" port: 45955 } } peers { permanent_uuid: "71d95c905a8f4ff181c1de9655d4a593" member_type: VOTER last_known_addr { host: "127.3.131.130" port: 35573 } } peers { permanent_uuid: "8571114a663448e99821d16f67d38bcb" member_type: VOTER last_known_addr { host: "127.3.131.131" port: 43851 } }
I20260621 05:10:27.903065 5337 catalog_manager.cc:5495] T 8f4d1fd6d2944817a641f0492c74cf07 P 71d95c905a8f4ff181c1de9655d4a593 reported cstate change: term changed from 0 to 1, leader changed from <none> to 71d95c905a8f4ff181c1de9655d4a593 (127.3.131.130). New cstate: current_term: 1 leader_uuid: "71d95c905a8f4ff181c1de9655d4a593" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f4e2cd7408443c0a14c51ebb038ec73" member_type: VOTER last_known_addr { host: "127.3.131.129" port: 45955 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "71d95c905a8f4ff181c1de9655d4a593" member_type: VOTER last_known_addr { host: "127.3.131.130" port: 35573 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8571114a663448e99821d16f67d38bcb" member_type: VOTER last_known_addr { host: "127.3.131.131" port: 43851 } health_report { overall_health: UNKNOWN } } }
I20260621 05:10:27.920996 5943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:10:27.918157 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48486 (local address 127.3.131.130:35573)
0621 05:10:27.918304 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0621 05:10:27.918306 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:10:27.918314 (+ 8us) server_negotiation.cc:407] Connection header received
0621 05:10:27.918340 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:10:27.918345 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:10:27.918357 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:10:27.918417 (+ 60us) server_negotiation.cc:226] Negotiated authn=TOKEN
0621 05:10:27.918632 (+ 215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:10:27.919463 (+ 831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:10:27.920254 (+ 791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:10:27.920467 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:10:27.920547 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:10:27.920716 (+ 169us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0621 05:10:27.920746 (+ 30us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0621 05:10:27.920792 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0621 05:10:27.920866 (+ 74us) server_negotiation.cc:299] Negotiation successful
0621 05:10:27.920885 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":122,"thread_start_us":100,"threads_started":1}
I20260621 05:10:27.926029 5943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:10:27.923617 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.131.130:34493 (local address 127.3.131.130:35573)
0621 05:10:27.923693 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0621 05:10:27.923695 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:10:27.923789 (+ 94us) server_negotiation.cc:407] Connection header received
0621 05:10:27.923827 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:10:27.923828 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:10:27.923837 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:10:27.923891 (+ 54us) server_negotiation.cc:226] Negotiated authn=TOKEN
0621 05:10:27.923973 (+ 82us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:10:27.924466 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:10:27.925303 (+ 837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:10:27.925487 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:10:27.925550 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:10:27.925737 (+ 187us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0621 05:10:27.925777 (+ 40us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0621 05:10:27.925828 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0621 05:10:27.925930 (+ 102us) server_negotiation.cc:299] Negotiation successful
0621 05:10:27.925954 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260621 05:10:27.926033 5946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:10:27.923555 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.131.130:35573 (local address 127.3.131.130:34493)
0621 05:10:27.923696 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0621 05:10:27.923707 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0621 05:10:27.923786 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:10:27.923898 (+ 112us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:10:27.923899 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:10:27.923900 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0621 05:10:27.923938 (+ 38us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:10:27.923940 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:10:27.924572 (+ 632us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:10:27.924573 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:10:27.925184 (+ 611us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:10:27.925186 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:10:27.925589 (+ 403us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:10:27.925590 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:10:27.925635 (+ 45us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:10:27.925639 (+ 4us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0621 05:10:27.925844 (+ 205us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0621 05:10:27.925847 (+ 3us) client_negotiation.cc:769] Sending connection context
0621 05:10:27.925888 (+ 41us) client_negotiation.cc:240] Negotiation successful
0621 05:10:27.925928 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":102,"threads_started":1}
I20260621 05:10:27.928503 5496 raft_consensus.cc:1261] T 8f4d1fd6d2944817a641f0492c74cf07 P 1f4e2cd7408443c0a14c51ebb038ec73 [term 1 FOLLOWER]: Refusing update from remote peer 71d95c905a8f4ff181c1de9655d4a593: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 05:10:27.928539 5776 raft_consensus.cc:1261] T 8f4d1fd6d2944817a641f0492c74cf07 P 8571114a663448e99821d16f67d38bcb [term 1 FOLLOWER]: Refusing update from remote peer 71d95c905a8f4ff181c1de9655d4a593: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 05:10:27.928791 5909 consensus_queue.cc:1035] T 8f4d1fd6d2944817a641f0492c74cf07 P 71d95c905a8f4ff181c1de9655d4a593 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f4e2cd7408443c0a14c51ebb038ec73" member_type: VOTER last_known_addr { host: "127.3.131.129" port: 45955 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 05:10:27.928905 5904 consensus_queue.cc:1035] T 8f4d1fd6d2944817a641f0492c74cf07 P 71d95c905a8f4ff181c1de9655d4a593 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8571114a663448e99821d16f67d38bcb" member_type: VOTER last_known_addr { host: "127.3.131.131" port: 43851 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 05:10:27.940201 5338 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:39504:
table { table_name: "test-table" } modify_external_catalogs: true
I20260621 05:10:27.940340 5338 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:39504:
table { table_name: "test-table" } modify_external_catalogs: true
I20260621 05:10:27.941062 5338 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b7017628c83e4962b0fb4f3348657105: Sending DeleteTablet for 3 replicas of tablet 8f4d1fd6d2944817a641f0492c74cf07
I20260621 05:10:27.941363 5476 tablet_service.cc:1514] Processing DeleteTablet for tablet 8f4d1fd6d2944817a641f0492c74cf07 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 05:10:27 UTC) from {username='kudu', principal='kudu/127.3.131.190@KRBTEST.COM'} at 127.0.0.1:47102
I20260621 05:10:27.941403 5615 tablet_service.cc:1514] Processing DeleteTablet for tablet 8f4d1fd6d2944817a641f0492c74cf07 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 05:10:27 UTC) from {username='kudu', principal='kudu/127.3.131.190@KRBTEST.COM'} at 127.0.0.1:48470
I20260621 05:10:27.941478 5923 tablet_replica.cc:330] T 8f4d1fd6d2944817a641f0492c74cf07 P 1f4e2cd7408443c0a14c51ebb038ec73: stopping tablet replica
I20260621 05:10:27.941538 5923 raft_consensus.cc:2229] T 8f4d1fd6d2944817a641f0492c74cf07 P 1f4e2cd7408443c0a14c51ebb038ec73 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260621 05:10:27.941550 5756 tablet_service.cc:1514] Processing DeleteTablet for tablet 8f4d1fd6d2944817a641f0492c74cf07 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 05:10:27 UTC) from {username='kudu', principal='kudu/127.3.131.190@KRBTEST.COM'} at 127.0.0.1:51708
I20260621 05:10:27.941582 5923 raft_consensus.cc:2258] T 8f4d1fd6d2944817a641f0492c74cf07 P 1f4e2cd7408443c0a14c51ebb038ec73 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 05:10:27.941818 5925 tablet_replica.cc:330] T 8f4d1fd6d2944817a641f0492c74cf07 P 8571114a663448e99821d16f67d38bcb: stopping tablet replica
I20260621 05:10:27.941648 5924 tablet_replica.cc:330] T 8f4d1fd6d2944817a641f0492c74cf07 P 71d95c905a8f4ff181c1de9655d4a593: stopping tablet replica
I20260621 05:10:27.941946 5923 ts_tablet_manager.cc:1905] T 8f4d1fd6d2944817a641f0492c74cf07 P 1f4e2cd7408443c0a14c51ebb038ec73: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 05:10:27.941968 5924 raft_consensus.cc:2229] T 8f4d1fd6d2944817a641f0492c74cf07 P 71d95c905a8f4ff181c1de9655d4a593 [term 1 LEADER]: Raft consensus shutting down.
I20260621 05:10:27.942073 5924 raft_consensus.cc:2258] T 8f4d1fd6d2944817a641f0492c74cf07 P 71d95c905a8f4ff181c1de9655d4a593 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 05:10:27.942173 5925 raft_consensus.cc:2229] T 8f4d1fd6d2944817a641f0492c74cf07 P 8571114a663448e99821d16f67d38bcb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260621 05:10:27.942221 5925 raft_consensus.cc:2258] T 8f4d1fd6d2944817a641f0492c74cf07 P 8571114a663448e99821d16f67d38bcb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 05:10:27.942601 5925 ts_tablet_manager.cc:1905] T 8f4d1fd6d2944817a641f0492c74cf07 P 8571114a663448e99821d16f67d38bcb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 05:10:27.942642 5924 ts_tablet_manager.cc:1905] T 8f4d1fd6d2944817a641f0492c74cf07 P 71d95c905a8f4ff181c1de9655d4a593: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 05:10:27.943682 5923 ts_tablet_manager.cc:1918] T 8f4d1fd6d2944817a641f0492c74cf07 P 1f4e2cd7408443c0a14c51ebb038ec73: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260621 05:10:27.943746 5923 log.cc:1196] T 8f4d1fd6d2944817a641f0492c74cf07 P 1f4e2cd7408443c0a14c51ebb038ec73: Deleting WAL directory at /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782018616648578-3598-0/minicluster-data/ts-0/wal/wals/8f4d1fd6d2944817a641f0492c74cf07
I20260621 05:10:27.943957 5923 ts_tablet_manager.cc:1939] T 8f4d1fd6d2944817a641f0492c74cf07 P 1f4e2cd7408443c0a14c51ebb038ec73: Deleting consensus metadata
I20260621 05:10:27.944214 5924 ts_tablet_manager.cc:1918] T 8f4d1fd6d2944817a641f0492c74cf07 P 71d95c905a8f4ff181c1de9655d4a593: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260621 05:10:27.944269 5924 log.cc:1196] T 8f4d1fd6d2944817a641f0492c74cf07 P 71d95c905a8f4ff181c1de9655d4a593: Deleting WAL directory at /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782018616648578-3598-0/minicluster-data/ts-1/wal/wals/8f4d1fd6d2944817a641f0492c74cf07
I20260621 05:10:27.944320 5338 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:39504:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260621 05:10:27.944497 5338 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260621 05:10:27.944522 5924 ts_tablet_manager.cc:1939] T 8f4d1fd6d2944817a641f0492c74cf07 P 71d95c905a8f4ff181c1de9655d4a593: Deleting consensus metadata
I20260621 05:10:27.944705 5925 ts_tablet_manager.cc:1918] T 8f4d1fd6d2944817a641f0492c74cf07 P 8571114a663448e99821d16f67d38bcb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260621 05:10:27.944757 5925 log.cc:1196] T 8f4d1fd6d2944817a641f0492c74cf07 P 8571114a663448e99821d16f67d38bcb: Deleting WAL directory at /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782018616648578-3598-0/minicluster-data/ts-2/wal/wals/8f4d1fd6d2944817a641f0492c74cf07
I20260621 05:10:27.944976 5925 ts_tablet_manager.cc:1939] T 8f4d1fd6d2944817a641f0492c74cf07 P 8571114a663448e99821d16f67d38bcb: Deleting consensus metadata
I20260621 05:10:27.945334 5325 catalog_manager.cc:4841] TS 8571114a663448e99821d16f67d38bcb (127.3.131.131:43851): tablet 8f4d1fd6d2944817a641f0492c74cf07 (table test-table [id=40c1f47a698c467087d6e37260b7ecea]) successfully deleted
I20260621 05:10:27.945415 5325 catalog_manager.cc:4841] TS 1f4e2cd7408443c0a14c51ebb038ec73 (127.3.131.129:45955): tablet 8f4d1fd6d2944817a641f0492c74cf07 (table test-table [id=40c1f47a698c467087d6e37260b7ecea]) successfully deleted
I20260621 05:10:27.945797 5324 catalog_manager.cc:4841] TS 71d95c905a8f4ff181c1de9655d4a593 (127.3.131.130:35573): tablet 8f4d1fd6d2944817a641f0492c74cf07 (table test-table [id=40c1f47a698c467087d6e37260b7ecea]) successfully deleted
I20260621 05:10:27.945878 5476 tablet_service.cc:1467] Processing CreateTablet for tablet a3df0b2a778b4b7eac45a80eac47d3bc (DEFAULT_TABLE table=test-table [id=0a9f8dffb3c54cc2879ebd95973b0d54]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 05:10:27.945878 5756 tablet_service.cc:1467] Processing CreateTablet for tablet a3df0b2a778b4b7eac45a80eac47d3bc (DEFAULT_TABLE table=test-table [id=0a9f8dffb3c54cc2879ebd95973b0d54]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 05:10:27.945981 5476 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a3df0b2a778b4b7eac45a80eac47d3bc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 05:10:27.945981 5756 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a3df0b2a778b4b7eac45a80eac47d3bc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 05:10:27.947124 5938 tablet_bootstrap.cc:492] T a3df0b2a778b4b7eac45a80eac47d3bc P 8571114a663448e99821d16f67d38bcb: Bootstrap starting.
I20260621 05:10:27.947149 5934 tablet_bootstrap.cc:492] T a3df0b2a778b4b7eac45a80eac47d3bc P 1f4e2cd7408443c0a14c51ebb038ec73: Bootstrap starting.
I20260621 05:10:27.947434 5615 tablet_service.cc:1467] Processing CreateTablet for tablet a3df0b2a778b4b7eac45a80eac47d3bc (DEFAULT_TABLE table=test-table [id=0a9f8dffb3c54cc2879ebd95973b0d54]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 05:10:27.947513 5934 tablet_bootstrap.cc:654] T a3df0b2a778b4b7eac45a80eac47d3bc P 1f4e2cd7408443c0a14c51ebb038ec73: Neither blocks nor log segments found. Creating new log.
I20260621 05:10:27.947548 5615 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a3df0b2a778b4b7eac45a80eac47d3bc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 05:10:27.947618 5938 tablet_bootstrap.cc:654] T a3df0b2a778b4b7eac45a80eac47d3bc P 8571114a663448e99821d16f67d38bcb: Neither blocks nor log segments found. Creating new log.
I20260621 05:10:27.948333 5938 tablet_bootstrap.cc:492] T a3df0b2a778b4b7eac45a80eac47d3bc P 8571114a663448e99821d16f67d38bcb: No bootstrap required, opened a new log
I20260621 05:10:27.948379 5938 ts_tablet_manager.cc:1397] T a3df0b2a778b4b7eac45a80eac47d3bc P 8571114a663448e99821d16f67d38bcb: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260621 05:10:27.948542 5938 raft_consensus.cc:348] T a3df0b2a778b4b7eac45a80eac47d3bc P 8571114a663448e99821d16f67d38bcb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f4e2cd7408443c0a14c51ebb038ec73" member_type: VOTER last_known_addr { host: "127.3.131.129" port: 45955 } } peers { permanent_uuid: "71d95c905a8f4ff181c1de9655d4a593" member_type: VOTER last_known_addr { host: "127.3.131.130" port: 35573 } } peers { permanent_uuid: "8571114a663448e99821d16f67d38bcb" member_type: VOTER last_known_addr { host: "127.3.131.131" port: 43851 } }
I20260621 05:10:27.948608 5938 raft_consensus.cc:374] T a3df0b2a778b4b7eac45a80eac47d3bc P 8571114a663448e99821d16f67d38bcb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 05:10:27.948628 5938 raft_consensus.cc:729] T a3df0b2a778b4b7eac45a80eac47d3bc P 8571114a663448e99821d16f67d38bcb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8571114a663448e99821d16f67d38bcb, State: Initialized, Role: FOLLOWER
I20260621 05:10:27.948673 5938 consensus_queue.cc:260] T a3df0b2a778b4b7eac45a80eac47d3bc P 8571114a663448e99821d16f67d38bcb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f4e2cd7408443c0a14c51ebb038ec73" member_type: VOTER last_known_addr { host: "127.3.131.129" port: 45955 } } peers { permanent_uuid: "71d95c905a8f4ff181c1de9655d4a593" member_type: VOTER last_known_addr { host: "127.3.131.130" port: 35573 } } peers { permanent_uuid: "8571114a663448e99821d16f67d38bcb" member_type: VOTER last_known_addr { host: "127.3.131.131" port: 43851 } }
I20260621 05:10:27.948774 5938 ts_tablet_manager.cc:1428] T a3df0b2a778b4b7eac45a80eac47d3bc P 8571114a663448e99821d16f67d38bcb: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260621 05:10:27.949009 5935 tablet_bootstrap.cc:492] T a3df0b2a778b4b7eac45a80eac47d3bc P 71d95c905a8f4ff181c1de9655d4a593: Bootstrap starting.
I20260621 05:10:27.949331 5935 tablet_bootstrap.cc:654] T a3df0b2a778b4b7eac45a80eac47d3bc P 71d95c905a8f4ff181c1de9655d4a593: Neither blocks nor log segments found. Creating new log.
I20260621 05:10:27.949819 5934 tablet_bootstrap.cc:492] T a3df0b2a778b4b7eac45a80eac47d3bc P 1f4e2cd7408443c0a14c51ebb038ec73: No bootstrap required, opened a new log
I20260621 05:10:27.949877 5934 ts_tablet_manager.cc:1397] T a3df0b2a778b4b7eac45a80eac47d3bc P 1f4e2cd7408443c0a14c51ebb038ec73: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.001s
I20260621 05:10:27.949995 5934 raft_consensus.cc:348] T a3df0b2a778b4b7eac45a80eac47d3bc P 1f4e2cd7408443c0a14c51ebb038ec73 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f4e2cd7408443c0a14c51ebb038ec73" member_type: VOTER last_known_addr { host: "127.3.131.129" port: 45955 } } peers { permanent_uuid: "71d95c905a8f4ff181c1de9655d4a593" member_type: VOTER last_known_addr { host: "127.3.131.130" port: 35573 } } peers { permanent_uuid: "8571114a663448e99821d16f67d38bcb" member_type: VOTER last_known_addr { host: "127.3.131.131" port: 43851 } }
I20260621 05:10:27.950071 5934 raft_consensus.cc:374] T a3df0b2a778b4b7eac45a80eac47d3bc P 1f4e2cd7408443c0a14c51ebb038ec73 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 05:10:27.950091 5934 raft_consensus.cc:729] T a3df0b2a778b4b7eac45a80eac47d3bc P 1f4e2cd7408443c0a14c51ebb038ec73 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f4e2cd7408443c0a14c51ebb038ec73, State: Initialized, Role: FOLLOWER
I20260621 05:10:27.950131 5934 consensus_queue.cc:260] T a3df0b2a778b4b7eac45a80eac47d3bc P 1f4e2cd7408443c0a14c51ebb038ec73 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f4e2cd7408443c0a14c51ebb038ec73" member_type: VOTER last_known_addr { host: "127.3.131.129" port: 45955 } } peers { permanent_uuid: "71d95c905a8f4ff181c1de9655d4a593" member_type: VOTER last_known_addr { host: "127.3.131.130" port: 35573 } } peers { permanent_uuid: "8571114a663448e99821d16f67d38bcb" member_type: VOTER last_known_addr { host: "127.3.131.131" port: 43851 } }
I20260621 05:10:27.950220 5934 ts_tablet_manager.cc:1428] T a3df0b2a778b4b7eac45a80eac47d3bc P 1f4e2cd7408443c0a14c51ebb038ec73: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260621 05:10:27.950554 5935 tablet_bootstrap.cc:492] T a3df0b2a778b4b7eac45a80eac47d3bc P 71d95c905a8f4ff181c1de9655d4a593: No bootstrap required, opened a new log
I20260621 05:10:27.950611 5935 ts_tablet_manager.cc:1397] T a3df0b2a778b4b7eac45a80eac47d3bc P 71d95c905a8f4ff181c1de9655d4a593: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260621 05:10:27.950747 5935 raft_consensus.cc:348] T a3df0b2a778b4b7eac45a80eac47d3bc P 71d95c905a8f4ff181c1de9655d4a593 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f4e2cd7408443c0a14c51ebb038ec73" member_type: VOTER last_known_addr { host: "127.3.131.129" port: 45955 } } peers { permanent_uuid: "71d95c905a8f4ff181c1de9655d4a593" member_type: VOTER last_known_addr { host: "127.3.131.130" port: 35573 } } peers { permanent_uuid: "8571114a663448e99821d16f67d38bcb" member_type: VOTER last_known_addr { host: "127.3.131.131" port: 43851 } }
I20260621 05:10:27.950819 5935 raft_consensus.cc:374] T a3df0b2a778b4b7eac45a80eac47d3bc P 71d95c905a8f4ff181c1de9655d4a593 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 05:10:27.950839 5935 raft_consensus.cc:729] T a3df0b2a778b4b7eac45a80eac47d3bc P 71d95c905a8f4ff181c1de9655d4a593 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 71d95c905a8f4ff181c1de9655d4a593, State: Initialized, Role: FOLLOWER
I20260621 05:10:27.950881 5935 consensus_queue.cc:260] T a3df0b2a778b4b7eac45a80eac47d3bc P 71d95c905a8f4ff181c1de9655d4a593 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f4e2cd7408443c0a14c51ebb038ec73" member_type: VOTER last_known_addr { host: "127.3.131.129" port: 45955 } } peers { permanent_uuid: "71d95c905a8f4ff181c1de9655d4a593" member_type: VOTER last_known_addr { host: "127.3.131.130" port: 35573 } } peers { permanent_uuid: "8571114a663448e99821d16f67d38bcb" member_type: VOTER last_known_addr { host: "127.3.131.131" port: 43851 } }
I20260621 05:10:27.950968 5935 ts_tablet_manager.cc:1428] T a3df0b2a778b4b7eac45a80eac47d3bc P 71d95c905a8f4ff181c1de9655d4a593: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260621 05:10:28.055547 5940 raft_consensus.cc:482] T a3df0b2a778b4b7eac45a80eac47d3bc P 8571114a663448e99821d16f67d38bcb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260621 05:10:28.055667 5940 raft_consensus.cc:504] T a3df0b2a778b4b7eac45a80eac47d3bc P 8571114a663448e99821d16f67d38bcb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f4e2cd7408443c0a14c51ebb038ec73" member_type: VOTER last_known_addr { host: "127.3.131.129" port: 45955 } } peers { permanent_uuid: "71d95c905a8f4ff181c1de9655d4a593" member_type: VOTER last_known_addr { host: "127.3.131.130" port: 35573 } } peers { permanent_uuid: "8571114a663448e99821d16f67d38bcb" member_type: VOTER last_known_addr { host: "127.3.131.131" port: 43851 } }
I20260621 05:10:28.056041 5940 leader_election.cc:288] T a3df0b2a778b4b7eac45a80eac47d3bc P 8571114a663448e99821d16f67d38bcb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1f4e2cd7408443c0a14c51ebb038ec73 (127.3.131.129:45955), 71d95c905a8f4ff181c1de9655d4a593 (127.3.131.130:35573)
I20260621 05:10:28.061174 5943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:10:28.056474 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.131.131:56147 (local address 127.3.131.130:35573)
0621 05:10:28.056521 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0621 05:10:28.056524 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0621 05:10:28.056621 (+ 97us) server_negotiation.cc:407] Connection header received
0621 05:10:28.056712 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:10:28.056713 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:10:28.056724 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:10:28.056787 (+ 63us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 05:10:28.057165 (+ 378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:10:28.057925 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:10:28.060034 (+ 2109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:10:28.060631 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:10:28.060745 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:10:28.060755 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0621 05:10:28.061043 (+ 288us) server_negotiation.cc:299] Negotiation successful
0621 05:10:28.061094 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260621 05:10:28.061218 5954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:10:28.056386 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.131.131:53291 (local address 127.3.131.129:45955)
0621 05:10:28.056522 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0621 05:10:28.056524 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:10:28.056605 (+ 81us) server_negotiation.cc:407] Connection header received
0621 05:10:28.056686 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:10:28.056687 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:10:28.056699 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:10:28.056788 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 05:10:28.057119 (+ 331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:10:28.057961 (+ 842us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:10:28.060035 (+ 2074us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:10:28.060671 (+ 636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:10:28.060745 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:10:28.060755 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0621 05:10:28.061043 (+ 288us) server_negotiation.cc:299] Negotiation successful
0621 05:10:28.061101 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":106,"thread_start_us":60,"threads_started":1}
I20260621 05:10:28.061275 5955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:10:28.056308 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.131.129:45955 (local address 127.3.131.131:53291)
0621 05:10:28.056532 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0621 05:10:28.056545 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0621 05:10:28.056622 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:10:28.056809 (+ 187us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:10:28.056810 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:10:28.056815 (+ 5us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 05:10:28.056903 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:10:28.056907 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:10:28.058091 (+ 1184us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:10:28.058092 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:10:28.059817 (+ 1725us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:10:28.059821 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:10:28.060798 (+ 977us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:10:28.060800 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:10:28.060920 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:10:28.060928 (+ 8us) client_negotiation.cc:769] Sending connection context
0621 05:10:28.061009 (+ 81us) client_negotiation.cc:240] Negotiation successful
0621 05:10:28.061072 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":196,"mutex_wait_us":92,"thread_start_us":82,"threads_started":1}
I20260621 05:10:28.061497 5953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:10:28.056384 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.131.130:35573 (local address 127.3.131.131:56147)
0621 05:10:28.056533 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0621 05:10:28.056545 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0621 05:10:28.056622 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:10:28.056809 (+ 187us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:10:28.056810 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:10:28.056813 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 05:10:28.056903 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:10:28.056906 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:10:28.058068 (+ 1162us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:10:28.058070 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:10:28.059816 (+ 1746us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:10:28.059821 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:10:28.060800 (+ 979us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:10:28.060801 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:10:28.060920 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:10:28.060930 (+ 10us) client_negotiation.cc:769] Sending connection context
0621 05:10:28.061009 (+ 79us) client_negotiation.cc:240] Negotiation successful
0621 05:10:28.061072 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":141,"mutex_wait_us":136,"thread_start_us":92,"threads_started":1}
I20260621 05:10:28.061635 5496 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3df0b2a778b4b7eac45a80eac47d3bc" candidate_uuid: "8571114a663448e99821d16f67d38bcb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f4e2cd7408443c0a14c51ebb038ec73" is_pre_election: true
I20260621 05:10:28.061786 5496 raft_consensus.cc:2454] T a3df0b2a778b4b7eac45a80eac47d3bc P 1f4e2cd7408443c0a14c51ebb038ec73 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8571114a663448e99821d16f67d38bcb in term 0.
I20260621 05:10:28.061792 5636 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3df0b2a778b4b7eac45a80eac47d3bc" candidate_uuid: "8571114a663448e99821d16f67d38bcb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "71d95c905a8f4ff181c1de9655d4a593" is_pre_election: true
I20260621 05:10:28.061882 5636 raft_consensus.cc:2454] T a3df0b2a778b4b7eac45a80eac47d3bc P 71d95c905a8f4ff181c1de9655d4a593 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8571114a663448e99821d16f67d38bcb in term 0.
I20260621 05:10:28.062071 5709 leader_election.cc:302] T a3df0b2a778b4b7eac45a80eac47d3bc P 8571114a663448e99821d16f67d38bcb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 1f4e2cd7408443c0a14c51ebb038ec73, 71d95c905a8f4ff181c1de9655d4a593, 8571114a663448e99821d16f67d38bcb; no voters:
I20260621 05:10:28.062297 5940 raft_consensus.cc:2781] T a3df0b2a778b4b7eac45a80eac47d3bc P 8571114a663448e99821d16f67d38bcb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260621 05:10:28.062346 5940 raft_consensus.cc:482] T a3df0b2a778b4b7eac45a80eac47d3bc P 8571114a663448e99821d16f67d38bcb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260621 05:10:28.062368 5940 raft_consensus.cc:3037] T a3df0b2a778b4b7eac45a80eac47d3bc P 8571114a663448e99821d16f67d38bcb [term 0 FOLLOWER]: Advancing to term 1
I20260621 05:10:28.063128 5940 raft_consensus.cc:504] T a3df0b2a778b4b7eac45a80eac47d3bc P 8571114a663448e99821d16f67d38bcb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f4e2cd7408443c0a14c51ebb038ec73" member_type: VOTER last_known_addr { host: "127.3.131.129" port: 45955 } } peers { permanent_uuid: "71d95c905a8f4ff181c1de9655d4a593" member_type: VOTER last_known_addr { host: "127.3.131.130" port: 35573 } } peers { permanent_uuid: "8571114a663448e99821d16f67d38bcb" member_type: VOTER last_known_addr { host: "127.3.131.131" port: 43851 } }
I20260621 05:10:28.063364 5940 leader_election.cc:288] T a3df0b2a778b4b7eac45a80eac47d3bc P 8571114a663448e99821d16f67d38bcb [CANDIDATE]: Term 1 election: Requested vote from peers 1f4e2cd7408443c0a14c51ebb038ec73 (127.3.131.129:45955), 71d95c905a8f4ff181c1de9655d4a593 (127.3.131.130:35573)
I20260621 05:10:28.063604 5496 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3df0b2a778b4b7eac45a80eac47d3bc" candidate_uuid: "8571114a663448e99821d16f67d38bcb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f4e2cd7408443c0a14c51ebb038ec73"
I20260621 05:10:28.063596 5636 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3df0b2a778b4b7eac45a80eac47d3bc" candidate_uuid: "8571114a663448e99821d16f67d38bcb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "71d95c905a8f4ff181c1de9655d4a593"
I20260621 05:10:28.063697 5496 raft_consensus.cc:3037] T a3df0b2a778b4b7eac45a80eac47d3bc P 1f4e2cd7408443c0a14c51ebb038ec73 [term 0 FOLLOWER]: Advancing to term 1
I20260621 05:10:28.063721 5636 raft_consensus.cc:3037] T a3df0b2a778b4b7eac45a80eac47d3bc P 71d95c905a8f4ff181c1de9655d4a593 [term 0 FOLLOWER]: Advancing to term 1
I20260621 05:10:28.064495 5636 raft_consensus.cc:2454] T a3df0b2a778b4b7eac45a80eac47d3bc P 71d95c905a8f4ff181c1de9655d4a593 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8571114a663448e99821d16f67d38bcb in term 1.
I20260621 05:10:28.064494 5496 raft_consensus.cc:2454] T a3df0b2a778b4b7eac45a80eac47d3bc P 1f4e2cd7408443c0a14c51ebb038ec73 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8571114a663448e99821d16f67d38bcb in term 1.
I20260621 05:10:28.064710 5708 leader_election.cc:302] T a3df0b2a778b4b7eac45a80eac47d3bc P 8571114a663448e99821d16f67d38bcb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 71d95c905a8f4ff181c1de9655d4a593, 8571114a663448e99821d16f67d38bcb; no voters:
I20260621 05:10:28.064831 5940 raft_consensus.cc:2781] T a3df0b2a778b4b7eac45a80eac47d3bc P 8571114a663448e99821d16f67d38bcb [term 1 FOLLOWER]: Leader election won for term 1
I20260621 05:10:28.065007 5940 raft_consensus.cc:686] T a3df0b2a778b4b7eac45a80eac47d3bc P 8571114a663448e99821d16f67d38bcb [term 1 LEADER]: Becoming Leader. State: Replica: 8571114a663448e99821d16f67d38bcb, State: Running, Role: LEADER
I20260621 05:10:28.065096 5940 consensus_queue.cc:237] T a3df0b2a778b4b7eac45a80eac47d3bc P 8571114a663448e99821d16f67d38bcb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f4e2cd7408443c0a14c51ebb038ec73" member_type: VOTER last_known_addr { host: "127.3.131.129" port: 45955 } } peers { permanent_uuid: "71d95c905a8f4ff181c1de9655d4a593" member_type: VOTER last_known_addr { host: "127.3.131.130" port: 35573 } } peers { permanent_uuid: "8571114a663448e99821d16f67d38bcb" member_type: VOTER last_known_addr { host: "127.3.131.131" port: 43851 } }
I20260621 05:10:28.065920 5338 catalog_manager.cc:5495] T a3df0b2a778b4b7eac45a80eac47d3bc P 8571114a663448e99821d16f67d38bcb reported cstate change: term changed from 0 to 1, leader changed from <none> to 8571114a663448e99821d16f67d38bcb (127.3.131.131). New cstate: current_term: 1 leader_uuid: "8571114a663448e99821d16f67d38bcb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f4e2cd7408443c0a14c51ebb038ec73" member_type: VOTER last_known_addr { host: "127.3.131.129" port: 45955 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "71d95c905a8f4ff181c1de9655d4a593" member_type: VOTER last_known_addr { host: "127.3.131.130" port: 35573 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8571114a663448e99821d16f67d38bcb" member_type: VOTER last_known_addr { host: "127.3.131.131" port: 43851 } health_report { overall_health: HEALTHY } } }
I20260621 05:10:28.100111 5958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:10:28.097455 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51720 (local address 127.3.131.131:43851)
0621 05:10:28.097647 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0621 05:10:28.097649 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:10:28.097657 (+ 8us) server_negotiation.cc:407] Connection header received
0621 05:10:28.097702 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:10:28.097703 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:10:28.097723 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:10:28.097817 (+ 94us) server_negotiation.cc:226] Negotiated authn=TOKEN
0621 05:10:28.097961 (+ 144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:10:28.098469 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:10:28.099239 (+ 770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:10:28.099466 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:10:28.099551 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:10:28.099730 (+ 179us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0621 05:10:28.099849 (+ 119us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0621 05:10:28.099930 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0621 05:10:28.099981 (+ 51us) server_negotiation.cc:299] Negotiation successful
0621 05:10:28.100003 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":165,"thread_start_us":112,"threads_started":1}
I20260621 05:10:28.105378 5943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:10:28.103112 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.131.131:48299 (local address 127.3.131.130:35573)
0621 05:10:28.103154 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0621 05:10:28.103156 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:10:28.103164 (+ 8us) server_negotiation.cc:407] Connection header received
0621 05:10:28.103217 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:10:28.103218 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:10:28.103231 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:10:28.103288 (+ 57us) server_negotiation.cc:226] Negotiated authn=TOKEN
0621 05:10:28.103421 (+ 133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:10:28.103897 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:10:28.104696 (+ 799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:10:28.104913 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:10:28.104978 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:10:28.105168 (+ 190us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0621 05:10:28.105196 (+ 28us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0621 05:10:28.105240 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0621 05:10:28.105297 (+ 57us) server_negotiation.cc:299] Negotiation successful
0621 05:10:28.105318 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260621 05:10:28.105410 5961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:10:28.102863 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.131.130:35573 (local address 127.3.131.131:48299)
0621 05:10:28.103045 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0621 05:10:28.103055 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0621 05:10:28.103096 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:10:28.103307 (+ 211us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:10:28.103308 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:10:28.103309 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0621 05:10:28.103361 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:10:28.103363 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:10:28.103977 (+ 614us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:10:28.103977 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:10:28.104592 (+ 615us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:10:28.104595 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:10:28.105022 (+ 427us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:10:28.105023 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:10:28.105074 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:10:28.105087 (+ 13us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0621 05:10:28.105258 (+ 171us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0621 05:10:28.105259 (+ 1us) client_negotiation.cc:769] Sending connection context
0621 05:10:28.105294 (+ 35us) client_negotiation.cc:240] Negotiation successful
0621 05:10:28.105333 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":96,"threads_started":1}
I20260621 05:10:28.107798 5496 raft_consensus.cc:1261] T a3df0b2a778b4b7eac45a80eac47d3bc P 1f4e2cd7408443c0a14c51ebb038ec73 [term 1 FOLLOWER]: Refusing update from remote peer 8571114a663448e99821d16f67d38bcb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 05:10:28.107820 5636 raft_consensus.cc:1261] T a3df0b2a778b4b7eac45a80eac47d3bc P 71d95c905a8f4ff181c1de9655d4a593 [term 1 FOLLOWER]: Refusing update from remote peer 8571114a663448e99821d16f67d38bcb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 05:10:28.108052 5940 consensus_queue.cc:1035] T a3df0b2a778b4b7eac45a80eac47d3bc P 8571114a663448e99821d16f67d38bcb [LEADER]: Connected to new peer: Peer: permanent_uuid: "71d95c905a8f4ff181c1de9655d4a593" member_type: VOTER last_known_addr { host: "127.3.131.130" port: 35573 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 05:10:28.108256 5956 consensus_queue.cc:1035] T a3df0b2a778b4b7eac45a80eac47d3bc P 8571114a663448e99821d16f67d38bcb [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f4e2cd7408443c0a14c51ebb038ec73" member_type: VOTER last_known_addr { host: "127.3.131.129" port: 45955 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 05:10:28.115918 5958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:10:28.113015 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.131.130:34817 (local address 127.3.131.131:43851)
0621 05:10:28.113167 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0621 05:10:28.113169 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:10:28.113176 (+ 7us) server_negotiation.cc:407] Connection header received
0621 05:10:28.113205 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:10:28.113206 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:10:28.113220 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:10:28.113316 (+ 96us) server_negotiation.cc:226] Negotiated authn=TOKEN
0621 05:10:28.113445 (+ 129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:10:28.113966 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:10:28.114774 (+ 808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:10:28.115151 (+ 377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:10:28.115312 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:10:28.115504 (+ 192us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0621 05:10:28.115545 (+ 41us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0621 05:10:28.115607 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0621 05:10:28.115842 (+ 235us) server_negotiation.cc:299] Negotiation successful
0621 05:10:28.115864 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":131}
I20260621 05:10:28.116029 5946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:10:28.112983 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.131.131:43851 (local address 127.3.131.130:34817)
0621 05:10:28.113040 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0621 05:10:28.113052 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0621 05:10:28.113093 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:10:28.113322 (+ 229us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:10:28.113323 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:10:28.113324 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0621 05:10:28.113372 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:10:28.113374 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:10:28.114031 (+ 657us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:10:28.114032 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:10:28.114680 (+ 648us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:10:28.114683 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:10:28.115309 (+ 626us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:10:28.115310 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:10:28.115386 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:10:28.115417 (+ 31us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0621 05:10:28.115679 (+ 262us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0621 05:10:28.115682 (+ 3us) client_negotiation.cc:769] Sending connection context
0621 05:10:28.115832 (+ 150us) client_negotiation.cc:240] Negotiation successful
0621 05:10:28.115934 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33}
W20260621 05:10:28.116464 5905 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260621 05:10:29.644323 5338 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:39504:
table { table_name: "test-table" } modify_external_catalogs: true
I20260621 05:10:29.644438 5338 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:39504:
table { table_name: "test-table" } modify_external_catalogs: true
I20260621 05:10:29.645470 5338 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b7017628c83e4962b0fb4f3348657105: Sending DeleteTablet for 3 replicas of tablet a3df0b2a778b4b7eac45a80eac47d3bc
I20260621 05:10:29.645831 5476 tablet_service.cc:1514] Processing DeleteTablet for tablet a3df0b2a778b4b7eac45a80eac47d3bc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 05:10:29 UTC) from {username='kudu', principal='kudu/127.3.131.190@KRBTEST.COM'} at 127.0.0.1:47102
I20260621 05:10:29.645987 3598 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWZOx2_/build/release/bin/kudu with pid 5411
I20260621 05:10:29.646502 5756 tablet_service.cc:1514] Processing DeleteTablet for tablet a3df0b2a778b4b7eac45a80eac47d3bc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 05:10:29 UTC) from {username='kudu', principal='kudu/127.3.131.190@KRBTEST.COM'} at 127.0.0.1:51708
I20260621 05:10:29.646732 5988 tablet_replica.cc:330] T a3df0b2a778b4b7eac45a80eac47d3bc P 8571114a663448e99821d16f67d38bcb: stopping tablet replica
I20260621 05:10:29.646723 5615 tablet_service.cc:1514] Processing DeleteTablet for tablet a3df0b2a778b4b7eac45a80eac47d3bc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 05:10:29 UTC) from {username='kudu', principal='kudu/127.3.131.190@KRBTEST.COM'} at 127.0.0.1:48470
I20260621 05:10:29.646812 5988 raft_consensus.cc:2229] T a3df0b2a778b4b7eac45a80eac47d3bc P 8571114a663448e99821d16f67d38bcb [term 1 LEADER]: Raft consensus shutting down.
I20260621 05:10:29.646940 5988 raft_consensus.cc:2258] T a3df0b2a778b4b7eac45a80eac47d3bc P 8571114a663448e99821d16f67d38bcb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 05:10:29.646947 5989 tablet_replica.cc:330] T a3df0b2a778b4b7eac45a80eac47d3bc P 71d95c905a8f4ff181c1de9655d4a593: stopping tablet replica
I20260621 05:10:29.647047 5989 raft_consensus.cc:2229] T a3df0b2a778b4b7eac45a80eac47d3bc P 71d95c905a8f4ff181c1de9655d4a593 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260621 05:10:29.647097 5989 raft_consensus.cc:2258] T a3df0b2a778b4b7eac45a80eac47d3bc P 71d95c905a8f4ff181c1de9655d4a593 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 05:10:29.647439 5988 ts_tablet_manager.cc:1905] T a3df0b2a778b4b7eac45a80eac47d3bc P 8571114a663448e99821d16f67d38bcb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 05:10:29.647459 5989 ts_tablet_manager.cc:1905] T a3df0b2a778b4b7eac45a80eac47d3bc P 71d95c905a8f4ff181c1de9655d4a593: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 05:10:29.649272 5988 ts_tablet_manager.cc:1918] T a3df0b2a778b4b7eac45a80eac47d3bc P 8571114a663448e99821d16f67d38bcb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260621 05:10:29.649264 5989 ts_tablet_manager.cc:1918] T a3df0b2a778b4b7eac45a80eac47d3bc P 71d95c905a8f4ff181c1de9655d4a593: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260621 05:10:29.649333 5988 log.cc:1196] T a3df0b2a778b4b7eac45a80eac47d3bc P 8571114a663448e99821d16f67d38bcb: Deleting WAL directory at /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782018616648578-3598-0/minicluster-data/ts-2/wal/wals/a3df0b2a778b4b7eac45a80eac47d3bc
I20260621 05:10:29.649333 5989 log.cc:1196] T a3df0b2a778b4b7eac45a80eac47d3bc P 71d95c905a8f4ff181c1de9655d4a593: Deleting WAL directory at /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782018616648578-3598-0/minicluster-data/ts-1/wal/wals/a3df0b2a778b4b7eac45a80eac47d3bc
I20260621 05:10:29.649562 5989 ts_tablet_manager.cc:1939] T a3df0b2a778b4b7eac45a80eac47d3bc P 71d95c905a8f4ff181c1de9655d4a593: Deleting consensus metadata
I20260621 05:10:29.649576 5988 ts_tablet_manager.cc:1939] T a3df0b2a778b4b7eac45a80eac47d3bc P 8571114a663448e99821d16f67d38bcb: Deleting consensus metadata
I20260621 05:10:29.650207 5324 catalog_manager.cc:4841] TS 71d95c905a8f4ff181c1de9655d4a593 (127.3.131.130:35573): tablet a3df0b2a778b4b7eac45a80eac47d3bc (table test-table [id=0a9f8dffb3c54cc2879ebd95973b0d54]) successfully deleted
I20260621 05:10:29.650251 5325 catalog_manager.cc:4841] TS 8571114a663448e99821d16f67d38bcb (127.3.131.131:43851): tablet a3df0b2a778b4b7eac45a80eac47d3bc (table test-table [id=0a9f8dffb3c54cc2879ebd95973b0d54]) successfully deleted
W20260621 05:10:29.654356 5325 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.3.131.129:45955 (error 108)
I20260621 05:10:29.654505 3598 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWZOx2_/build/release/bin/kudu with pid 5551
I20260621 05:10:29.656145 5991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:10:29.655900 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.131.129:45955 (local address 127.0.0.1:47116)
0621 05:10:29.656039 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0621 05:10:29.656090 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.131.129:45955: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":92,"threads_started":1}
W20260621 05:10:29.659420 5325 catalog_manager.cc:4584] TS 1f4e2cd7408443c0a14c51ebb038ec73 (127.3.131.129:45955): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet a3df0b2a778b4b7eac45a80eac47d3bc: Network error: Client connection negotiation failed: client connection to 127.3.131.129:45955: connect: Connection refused (error 111)
I20260621 05:10:29.661182 3598 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWZOx2_/build/release/bin/kudu with pid 5691
I20260621 05:10:29.667277 3598 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWZOx2_/build/release/bin/kudu with pid 5307
2026-06-21T05:10:29Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (5958 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782018616648578-3598-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 21 05:10:29 dist-test-slave-vrq1 krb5kdc[5994](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 21 05:10:29 dist-test-slave-vrq1 krb5kdc[5994](info): set up 2 sockets
Jun 21 05:10:29 dist-test-slave-vrq1 krb5kdc[5994](info): commencing operation
krb5kdc: starting...
W20260621 05:10:30.747512 3598 mini_kdc.cc:121] Time spent starting KDC: real 1.057s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782018616648578-3598-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782018616648578-3598-0/krb5kdc/test-user.keytab.
Jun 21 05:10:30 dist-test-slave-vrq1 krb5kdc[5994](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782018630, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-21T05:10:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-21T05:10:30Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.131.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.131.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.131.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782018616648578-3598-0/krb5kdc/kudu_127.3.131.190.keytab.
Entry for principal kudu/127.3.131.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782018616648578-3598-0/krb5kdc/kudu_127.3.131.190.keytab.
I20260621 05:10:30.881448 3598 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWZOx2_/build/release/bin/kudu
/tmp/dist-test-taskWZOx2_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782018616648578-3598-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782018616648578-3598-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782018616648578-3598-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782018616648578-3598-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.131.190:46269
--webserver_interface=127.3.131.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782018616648578-3598-0/krb5kdc/kudu_127.3.131.190.keytab
--principal=kudu/127.3.131.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.131.148:37207
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.131.190:46269
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782018616648578-3598-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782018616648578-3598-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782018616648578-3598-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 05:10:30.965193 6007 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 05:10:30.965546 6007 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 05:10:30.965572 6007 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 05:10:30.966872 6007 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 05:10:30.966922 6007 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 05:10:30.966938 6007 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 05:10:30.966950 6007 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 05:10:30.966964 6007 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 05:10:30.968585 6007 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.131.148:37207
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782018616648578-3598-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782018616648578-3598-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.131.190:46269
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782018616648578-3598-0/krb5kdc/kudu_127.3.131.190.keytab
--principal=kudu/127.3.131.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.131.190:46269
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782018616648578-3598-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.131.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6007
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782018616648578-3598-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 21 Jun 2026 04:16:16 UTC on ccycloud.quasar-xycaha.root.comops.site
build id 4457881
I20260621 05:10:30.968875 6007 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 05:10:30.969131 6007 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 05:10:30.974634 6012 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:10:30.974757 6013 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:10:30.974879 6007 server_base.cc:1034] running on GCE node
W20260621 05:10:30.974830 6015 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:10:30.975296 6007 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 05:10:30.975622 6007 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 05:10:30.976802 6007 hybrid_clock.cc:648] HybridClock initialized: now 1782018630976808 us; error 48 us; skew 500 ppm
Jun 21 05:10:30 dist-test-slave-vrq1 krb5kdc[5994](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782018630, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.131.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 05:10:30.979462 6007 init.cc:377] Logged in from keytab as kudu/127.3.131.190@KRBTEST.COM (short username kudu)
I20260621 05:10:30.979976 6007 webserver.cc:460] Webserver started at http://127.3.131.190:37839/ using document root <none> and password file <none>
I20260621 05:10:30.980219 6007 fs_manager.cc:359] Metadata directory not provided
I20260621 05:10:30.980289 6007 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 05:10:30.980443 6007 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 05:10:30.981523 6007 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782018616648578-3598-0/minicluster-data/master-0/data/instance:
uuid: "5ad2862814864b0796936b2261d9ec0b"
format_stamp: "Formatted at 2026-06-21 05:10:30 on dist-test-slave-vrq1"
server_key: "778e60d0c55bc4b089e08d5cfa9851a1"
server_key_iv: "2c1d00680fd531de962f970ac740433c"
server_key_version: "encryptionkey@0"
I20260621 05:10:30.981895 6007 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782018616648578-3598-0/minicluster-data/master-0/wal/instance:
uuid: "5ad2862814864b0796936b2261d9ec0b"
format_stamp: "Formatted at 2026-06-21 05:10:30 on dist-test-slave-vrq1"
server_key: "778e60d0c55bc4b089e08d5cfa9851a1"
server_key_iv: "2c1d00680fd531de962f970ac740433c"
server_key_version: "encryptionkey@0"
I20260621 05:10:30.983336 6007 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.001s
I20260621 05:10:30.984097 6022 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 05:10:30.984290 6007 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260621 05:10:30.984390 6007 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782018616648578-3598-0/minicluster-data/master-0/data,/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782018616648578-3598-0/minicluster-data/master-0/wal
uuid: "5ad2862814864b0796936b2261d9ec0b"
format_stamp: "Formatted at 2026-06-21 05:10:30 on dist-test-slave-vrq1"
server_key: "778e60d0c55bc4b089e08d5cfa9851a1"
server_key_iv: "2c1d00680fd531de962f970ac740433c"
server_key_version: "encryptionkey@0"
I20260621 05:10:30.984478 6007 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782018616648578-3598-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782018616648578-3598-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782018616648578-3598-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 05:10:31.000006 6007 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 05:10:31.002481 6007 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 05:10:31.002698 6007 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 05:10:31.007104 6007 rpc_server.cc:307] RPC server started. Bound to: 127.3.131.190:46269
I20260621 05:10:31.007133 6074 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.131.190:46269 every 8 connection(s)
I20260621 05:10:31.007552 6007 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782018616648578-3598-0/minicluster-data/master-0/data/info.pb
I20260621 05:10:31.008502 6075 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 05:10:31.010774 6075 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5ad2862814864b0796936b2261d9ec0b: Bootstrap starting.
I20260621 05:10:31.011418 6075 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5ad2862814864b0796936b2261d9ec0b: Neither blocks nor log segments found. Creating new log.
I20260621 05:10:31.011682 6075 log.cc:824] T 00000000000000000000000000000000 P 5ad2862814864b0796936b2261d9ec0b: Log is configured to *not* fsync() on all Append() calls
I20260621 05:10:31.012400 6075 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5ad2862814864b0796936b2261d9ec0b: No bootstrap required, opened a new log
I20260621 05:10:31.013996 6075 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5ad2862814864b0796936b2261d9ec0b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ad2862814864b0796936b2261d9ec0b" member_type: VOTER last_known_addr { host: "127.3.131.190" port: 46269 } }
I20260621 05:10:31.014145 6075 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5ad2862814864b0796936b2261d9ec0b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 05:10:31.014168 6075 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5ad2862814864b0796936b2261d9ec0b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5ad2862814864b0796936b2261d9ec0b, State: Initialized, Role: FOLLOWER
I20260621 05:10:31.014328 6075 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5ad2862814864b0796936b2261d9ec0b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ad2862814864b0796936b2261d9ec0b" member_type: VOTER last_known_addr { host: "127.3.131.190" port: 46269 } }
I20260621 05:10:31.014418 6075 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5ad2862814864b0796936b2261d9ec0b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260621 05:10:31.014442 6075 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5ad2862814864b0796936b2261d9ec0b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260621 05:10:31.014472 6075 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5ad2862814864b0796936b2261d9ec0b [term 0 FOLLOWER]: Advancing to term 1
I20260621 05:10:31.014981 6075 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5ad2862814864b0796936b2261d9ec0b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ad2862814864b0796936b2261d9ec0b" member_type: VOTER last_known_addr { host: "127.3.131.190" port: 46269 } }
I20260621 05:10:31.015066 6075 leader_election.cc:302] T 00000000000000000000000000000000 P 5ad2862814864b0796936b2261d9ec0b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5ad2862814864b0796936b2261d9ec0b; no voters:
I20260621 05:10:31.015285 6075 leader_election.cc:288] T 00000000000000000000000000000000 P 5ad2862814864b0796936b2261d9ec0b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260621 05:10:31.015379 6078 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5ad2862814864b0796936b2261d9ec0b [term 1 FOLLOWER]: Leader election won for term 1
I20260621 05:10:31.015580 6075 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5ad2862814864b0796936b2261d9ec0b [sys.catalog]: configured and running, proceeding with master startup.
I20260621 05:10:31.015808 6078 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5ad2862814864b0796936b2261d9ec0b [term 1 LEADER]: Becoming Leader. State: Replica: 5ad2862814864b0796936b2261d9ec0b, State: Running, Role: LEADER
I20260621 05:10:31.015934 6078 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5ad2862814864b0796936b2261d9ec0b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ad2862814864b0796936b2261d9ec0b" member_type: VOTER last_known_addr { host: "127.3.131.190" port: 46269 } }
I20260621 05:10:31.016465 6079 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5ad2862814864b0796936b2261d9ec0b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5ad2862814864b0796936b2261d9ec0b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ad2862814864b0796936b2261d9ec0b" member_type: VOTER last_known_addr { host: "127.3.131.190" port: 46269 } } }
I20260621 05:10:31.016559 6079 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5ad2862814864b0796936b2261d9ec0b [sys.catalog]: This master's current role is: LEADER
I20260621 05:10:31.016531 6080 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5ad2862814864b0796936b2261d9ec0b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5ad2862814864b0796936b2261d9ec0b. Latest consensus state: current_term: 1 leader_uuid: "5ad2862814864b0796936b2261d9ec0b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ad2862814864b0796936b2261d9ec0b" member_type: VOTER last_known_addr { host: "127.3.131.190" port: 46269 } } }
I20260621 05:10:31.016616 6080 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5ad2862814864b0796936b2261d9ec0b [sys.catalog]: This master's current role is: LEADER
I20260621 05:10:31.016829 6087 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260621 05:10:31.017261 3598 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWZOx2_/build/release/bin/kudu as pid 6007
I20260621 05:10:31.017333 3598 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782018616648578-3598-0/minicluster-data/master-0/wal/instance
I20260621 05:10:31.017517 3598 external_mini_cluster.cc:1428] Setting key 5da44afaef71ee9aa3caa776d0b27b8b
I20260621 05:10:31.018172 6087 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260621 05:10:31.019980 6087 catalog_manager.cc:1347] Generated new cluster ID: 1656a9bba5c144f7ade323007238ff6f
I20260621 05:10:31.020059 6087 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260621 05:10:31.025640 6087 catalog_manager.cc:1370] Generated new certificate authority record
I20260621 05:10:31.026382 6087 catalog_manager.cc:1504] Loading token signing keys...
I20260621 05:10:31.030181 6087 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5ad2862814864b0796936b2261d9ec0b: Generated new TSK 0
I20260621 05:10:31.030371 6087 catalog_manager.cc:1514] Initializing in-progress tserver states...
Jun 21 05:10:31 dist-test-slave-vrq1 krb5kdc[5994](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782018630, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.131.190@KRBTEST.COM
I20260621 05:10:31.035246 6096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:10:31.018014 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57128 (local address 127.3.131.190:46269)
0621 05:10:31.018217 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0621 05:10:31.018229 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0621 05:10:31.025443 (+ 7214us) server_negotiation.cc:407] Connection header received
0621 05:10:31.025627 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:10:31.025635 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:10:31.025745 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:10:31.025892 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:10:31.026578 (+ 686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:10:31.027596 (+ 1018us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:10:31.028362 (+ 766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:10:31.028821 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:10:31.028951 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:10:31.031939 (+ 2988us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:10:31.031943 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:10:31.031943 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:10:31.031955 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:10:31.033576 (+ 1621us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:10:31.033996 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:10:31.033999 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:10:31.033999 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:10:31.034103 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:10:31.034310 (+ 207us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:10:31.034311 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:10:31.034311 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:10:31.034546 (+ 235us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:10:31.034660 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0621 05:10:31.034802 (+ 142us) server_negotiation.cc:299] Negotiation successful
0621 05:10:31.034923 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":142,"thread_start_us":91,"threads_started":1}
WARNING: no policy specified for kudu/127.3.131.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.131.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.131.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782018616648578-3598-0/krb5kdc/kudu_127.3.131.129.keytab.
Entry for principal kudu/127.3.131.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782018616648578-3598-0/krb5kdc/kudu_127.3.131.129.keytab.
I20260621 05:10:31.063896 3598 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWZOx2_/build/release/bin/kudu
/tmp/dist-test-taskWZOx2_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782018616648578-3598-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782018616648578-3598-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782018616648578-3598-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782018616648578-3598-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.131.129:0
--local_ip_for_outbound_sockets=127.3.131.129
--webserver_interface=127.3.131.129
--webserver_port=0
--tserver_master_addrs=127.3.131.190:46269
--keytab_file=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782018616648578-3598-0/krb5kdc/kudu_127.3.131.129.keytab
--principal=kudu/127.3.131.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.131.148:37207
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782018616648578-3598-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782018616648578-3598-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782018616648578-3598-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 05:10:31.152757 6101 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 05:10:31.152979 6101 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 05:10:31.153007 6101 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 05:10:31.154282 6101 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 05:10:31.154332 6101 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 05:10:31.154363 6101 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.131.129
I20260621 05:10:31.156038 6101 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.131.148:37207
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782018616648578-3598-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782018616648578-3598-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782018616648578-3598-0/krb5kdc/kudu_127.3.131.129.keytab
--principal=kudu/127.3.131.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.131.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782018616648578-3598-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.131.129
--webserver_port=0
--tserver_master_addrs=127.3.131.190:46269
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6101
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.131.129
--log_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782018616648578-3598-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 21 Jun 2026 04:16:16 UTC on ccycloud.quasar-xycaha.root.comops.site
build id 4457881
I20260621 05:10:31.156324 6101 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 05:10:31.156597 6101 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 05:10:31.162225 6109 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:10:31.162158 6107 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:10:31.162175 6106 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:10:31.162505 6101 server_base.cc:1034] running on GCE node
I20260621 05:10:31.162771 6101 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 05:10:31.163044 6101 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 05:10:31.164216 6101 hybrid_clock.cc:648] HybridClock initialized: now 1782018631164204 us; error 53 us; skew 500 ppm
Jun 21 05:10:31 dist-test-slave-vrq1 krb5kdc[5994](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782018631, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.131.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 05:10:31.166972 6101 init.cc:377] Logged in from keytab as kudu/127.3.131.129@KRBTEST.COM (short username kudu)
I20260621 05:10:31.167493 6101 webserver.cc:460] Webserver started at http://127.3.131.129:41215/ using document root <none> and password file <none>
I20260621 05:10:31.167742 6101 fs_manager.cc:359] Metadata directory not provided
I20260621 05:10:31.167788 6101 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 05:10:31.167922 6101 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 05:10:31.169305 6101 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782018616648578-3598-0/minicluster-data/ts-0/data/instance:
uuid: "58d83d8bae1f4e6aa1db15339e9b11d6"
format_stamp: "Formatted at 2026-06-21 05:10:31 on dist-test-slave-vrq1"
server_key: "880aef877706fef015d804a0c9e5ce81"
server_key_iv: "f8ff73ad5025691dc41d88b88c0ee772"
server_key_version: "encryptionkey@0"
I20260621 05:10:31.169729 6101 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782018616648578-3598-0/minicluster-data/ts-0/wal/instance:
uuid: "58d83d8bae1f4e6aa1db15339e9b11d6"
format_stamp: "Formatted at 2026-06-21 05:10:31 on dist-test-slave-vrq1"
server_key: "880aef877706fef015d804a0c9e5ce81"
server_key_iv: "f8ff73ad5025691dc41d88b88c0ee772"
server_key_version: "encryptionkey@0"
I20260621 05:10:31.171298 6101 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.004s sys 0.000s
I20260621 05:10:31.172142 6116 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 05:10:31.172379 6101 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260621 05:10:31.172490 6101 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782018616648578-3598-0/minicluster-data/ts-0/data,/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782018616648578-3598-0/minicluster-data/ts-0/wal
uuid: "58d83d8bae1f4e6aa1db15339e9b11d6"
format_stamp: "Formatted at 2026-06-21 05:10:31 on dist-test-slave-vrq1"
server_key: "880aef877706fef015d804a0c9e5ce81"
server_key_iv: "f8ff73ad5025691dc41d88b88c0ee772"
server_key_version: "encryptionkey@0"
I20260621 05:10:31.172554 6101 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782018616648578-3598-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782018616648578-3598-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782018616648578-3598-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 05:10:31.200088 6101 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 05:10:31.202669 6101 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 05:10:31.202852 6101 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 05:10:31.203050 6101 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 05:10:31.203429 6101 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 05:10:31.203471 6101 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 05:10:31.203495 6101 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 05:10:31.203508 6101 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 05:10:31.209276 6101 rpc_server.cc:307] RPC server started. Bound to: 127.3.131.129:36947
I20260621 05:10:31.209350 6229 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.131.129:36947 every 8 connection(s)
I20260621 05:10:31.209761 6101 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782018616648578-3598-0/minicluster-data/ts-0/data/info.pb
Jun 21 05:10:31 dist-test-slave-vrq1 krb5kdc[5994](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782018631, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.131.129@KRBTEST.COM for kudu/127.3.131.190@KRBTEST.COM
I20260621 05:10:31.219846 3598 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWZOx2_/build/release/bin/kudu as pid 6101
I20260621 05:10:31.219956 3598 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782018616648578-3598-0/minicluster-data/ts-0/wal/instance
I20260621 05:10:31.220142 3598 external_mini_cluster.cc:1428] Setting key a220c5ad5d2cd4da3ff22e8ae3cfe4ab
I20260621 05:10:31.220688 6096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:10:31.210625 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.131.129:35549 (local address 127.3.131.190:46269)
0621 05:10:31.210706 (+ 81us) server_negotiation.cc:206] Beginning negotiation
0621 05:10:31.210708 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:10:31.210987 (+ 279us) server_negotiation.cc:407] Connection header received
0621 05:10:31.211508 (+ 521us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:10:31.211510 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:10:31.211521 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:10:31.211572 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:10:31.212660 (+ 1088us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:10:31.213230 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:10:31.214416 (+ 1186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:10:31.214629 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:10:31.214721 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:10:31.217681 (+ 2960us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:10:31.217683 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:10:31.217683 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:10:31.217691 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:10:31.219077 (+ 1386us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:10:31.219528 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:10:31.219529 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:10:31.219529 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:10:31.219585 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:10:31.219894 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:10:31.219895 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:10:31.219896 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:10:31.220072 (+ 176us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:10:31.220133 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0621 05:10:31.220499 (+ 366us) server_negotiation.cc:299] Negotiation successful
0621 05:10:31.220603 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260621 05:10:31.220937 6232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:10:31.210577 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.131.190:46269 (local address 127.3.131.129:35549)
0621 05:10:31.210803 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0621 05:10:31.210858 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0621 05:10:31.211403 (+ 545us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:10:31.211642 (+ 239us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:10:31.211645 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:10:31.212314 (+ 669us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:10:31.212533 (+ 219us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:10:31.212539 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:10:31.213362 (+ 823us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:10:31.213363 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:10:31.214309 (+ 946us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:10:31.214313 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:10:31.214741 (+ 428us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:10:31.214742 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:10:31.214866 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:10:31.215508 (+ 642us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 05:10:31.215517 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:10:31.217525 (+ 2008us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 05:10:31.219192 (+ 1667us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:10:31.219201 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:10:31.219204 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:10:31.219446 (+ 242us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:10:31.219646 (+ 200us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:10:31.219647 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:10:31.219647 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:10:31.219780 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:10:31.220136 (+ 356us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 05:10:31.220144 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 05:10:31.220350 (+ 206us) client_negotiation.cc:769] Sending connection context
0621 05:10:31.220460 (+ 110us) client_negotiation.cc:240] Negotiation successful
0621 05:10:31.220637 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":110,"threads_started":1}
I20260621 05:10:31.221436 6230 heartbeater.cc:347] Connected to a master server at 127.3.131.190:46269
I20260621 05:10:31.221541 6230 heartbeater.cc:464] Registering TS with master...
I20260621 05:10:31.221755 6230 heartbeater.cc:511] Master 127.3.131.190:46269 requested a full tablet report, sending...
I20260621 05:10:31.222215 6039 ts_manager.cc:194] Registered new tserver with Master: 58d83d8bae1f4e6aa1db15339e9b11d6 (127.3.131.129:36947)
I20260621 05:10:31.222867 6039 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.131.129@KRBTEST.COM'} at 127.3.131.129:35549
WARNING: no policy specified for kudu/127.3.131.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.131.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.131.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782018616648578-3598-0/krb5kdc/kudu_127.3.131.130.keytab.
Entry for principal kudu/127.3.131.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782018616648578-3598-0/krb5kdc/kudu_127.3.131.130.keytab.
I20260621 05:10:31.250967 3598 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWZOx2_/build/release/bin/kudu
/tmp/dist-test-taskWZOx2_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782018616648578-3598-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782018616648578-3598-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782018616648578-3598-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782018616648578-3598-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.131.130:0
--local_ip_for_outbound_sockets=127.3.131.130
--webserver_interface=127.3.131.130
--webserver_port=0
--tserver_master_addrs=127.3.131.190:46269
--keytab_file=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782018616648578-3598-0/krb5kdc/kudu_127.3.131.130.keytab
--principal=kudu/127.3.131.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.131.148:37207
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782018616648578-3598-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782018616648578-3598-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782018616648578-3598-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 05:10:31.340348 6235 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 05:10:31.340612 6235 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 05:10:31.340654 6235 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 05:10:31.341868 6235 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 05:10:31.341945 6235 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 05:10:31.341989 6235 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.131.130
I20260621 05:10:31.343554 6235 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.131.148:37207
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782018616648578-3598-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782018616648578-3598-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782018616648578-3598-0/krb5kdc/kudu_127.3.131.130.keytab
--principal=kudu/127.3.131.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.131.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782018616648578-3598-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.131.130
--webserver_port=0
--tserver_master_addrs=127.3.131.190:46269
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6235
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.131.130
--log_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782018616648578-3598-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 21 Jun 2026 04:16:16 UTC on ccycloud.quasar-xycaha.root.comops.site
build id 4457881
I20260621 05:10:31.343847 6235 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 05:10:31.344084 6235 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 05:10:31.349498 6240 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:10:31.349505 6241 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:10:31.349505 6243 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:10:31.349870 6235 server_base.cc:1034] running on GCE node
I20260621 05:10:31.350106 6235 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 05:10:31.350366 6235 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 05:10:31.351572 6235 hybrid_clock.cc:648] HybridClock initialized: now 1782018631351562 us; error 37 us; skew 500 ppm
Jun 21 05:10:31 dist-test-slave-vrq1 krb5kdc[5994](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782018631, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.131.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 05:10:31.354307 6235 init.cc:377] Logged in from keytab as kudu/127.3.131.130@KRBTEST.COM (short username kudu)
I20260621 05:10:31.354852 6235 webserver.cc:460] Webserver started at http://127.3.131.130:34735/ using document root <none> and password file <none>
I20260621 05:10:31.355114 6235 fs_manager.cc:359] Metadata directory not provided
I20260621 05:10:31.355207 6235 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 05:10:31.355381 6235 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 05:10:31.356573 6235 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782018616648578-3598-0/minicluster-data/ts-1/data/instance:
uuid: "9cc82f280100477a81e260f6e1056525"
format_stamp: "Formatted at 2026-06-21 05:10:31 on dist-test-slave-vrq1"
server_key: "fb40fb8499fd8bd792b7b659e6955905"
server_key_iv: "5d56243719a1d1dbcb8094e335deb666"
server_key_version: "encryptionkey@0"
I20260621 05:10:31.356961 6235 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782018616648578-3598-0/minicluster-data/ts-1/wal/instance:
uuid: "9cc82f280100477a81e260f6e1056525"
format_stamp: "Formatted at 2026-06-21 05:10:31 on dist-test-slave-vrq1"
server_key: "fb40fb8499fd8bd792b7b659e6955905"
server_key_iv: "5d56243719a1d1dbcb8094e335deb666"
server_key_version: "encryptionkey@0"
I20260621 05:10:31.358440 6235 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260621 05:10:31.359452 6250 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 05:10:31.359742 6235 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260621 05:10:31.359860 6235 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782018616648578-3598-0/minicluster-data/ts-1/data,/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782018616648578-3598-0/minicluster-data/ts-1/wal
uuid: "9cc82f280100477a81e260f6e1056525"
format_stamp: "Formatted at 2026-06-21 05:10:31 on dist-test-slave-vrq1"
server_key: "fb40fb8499fd8bd792b7b659e6955905"
server_key_iv: "5d56243719a1d1dbcb8094e335deb666"
server_key_version: "encryptionkey@0"
I20260621 05:10:31.359942 6235 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782018616648578-3598-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782018616648578-3598-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782018616648578-3598-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 05:10:31.372752 6235 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 05:10:31.375695 6235 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 05:10:31.375890 6235 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 05:10:31.376163 6235 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 05:10:31.376546 6235 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 05:10:31.376611 6235 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 05:10:31.376646 6235 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 05:10:31.376659 6235 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 05:10:31.382841 6235 rpc_server.cc:307] RPC server started. Bound to: 127.3.131.130:39577
I20260621 05:10:31.382948 6363 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.131.130:39577 every 8 connection(s)
I20260621 05:10:31.383287 6235 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782018616648578-3598-0/minicluster-data/ts-1/data/info.pb
I20260621 05:10:31.386931 3598 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWZOx2_/build/release/bin/kudu as pid 6235
I20260621 05:10:31.387041 3598 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782018616648578-3598-0/minicluster-data/ts-1/wal/instance
I20260621 05:10:31.387290 3598 external_mini_cluster.cc:1428] Setting key d16ad1aeb3d7a1fdb89d9c73ccbf732f
Jun 21 05:10:31 dist-test-slave-vrq1 krb5kdc[5994](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782018631, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.131.130@KRBTEST.COM for kudu/127.3.131.190@KRBTEST.COM
I20260621 05:10:31.394699 6096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:10:31.384012 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.131.130:51013 (local address 127.3.131.190:46269)
0621 05:10:31.384078 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0621 05:10:31.384080 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:10:31.384338 (+ 258us) server_negotiation.cc:407] Connection header received
0621 05:10:31.384831 (+ 493us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:10:31.384833 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:10:31.384848 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:10:31.384930 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:10:31.386215 (+ 1285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:10:31.386798 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:10:31.388289 (+ 1491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:10:31.388536 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:10:31.388594 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:10:31.391027 (+ 2433us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:10:31.391028 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:10:31.391029 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:10:31.391038 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:10:31.392937 (+ 1899us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:10:31.393459 (+ 522us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:10:31.393460 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:10:31.393461 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:10:31.393530 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:10:31.393799 (+ 269us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:10:31.393800 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:10:31.393800 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:10:31.393975 (+ 175us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:10:31.394057 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0621 05:10:31.394502 (+ 445us) server_negotiation.cc:299] Negotiation successful
0621 05:10:31.394597 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260621 05:10:31.394857 6366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:10:31.383999 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.131.190:46269 (local address 127.3.131.130:51013)
0621 05:10:31.384176 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0621 05:10:31.384220 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0621 05:10:31.384705 (+ 485us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:10:31.385096 (+ 391us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:10:31.385101 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:10:31.385817 (+ 716us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:10:31.386076 (+ 259us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:10:31.386083 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:10:31.386878 (+ 795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:10:31.386879 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:10:31.388199 (+ 1320us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:10:31.388202 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:10:31.388645 (+ 443us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:10:31.388646 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:10:31.388805 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:10:31.389448 (+ 643us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 05:10:31.389459 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:10:31.390927 (+ 1468us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 05:10:31.393081 (+ 2154us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:10:31.393089 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:10:31.393091 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:10:31.393370 (+ 279us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:10:31.393610 (+ 240us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:10:31.393610 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:10:31.393611 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:10:31.393727 (+ 116us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:10:31.394205 (+ 478us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 05:10:31.394212 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 05:10:31.394343 (+ 131us) client_negotiation.cc:769] Sending connection context
0621 05:10:31.394446 (+ 103us) client_negotiation.cc:240] Negotiation successful
0621 05:10:31.394584 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":98,"threads_started":1}
I20260621 05:10:31.395449 6364 heartbeater.cc:347] Connected to a master server at 127.3.131.190:46269
I20260621 05:10:31.395705 6364 heartbeater.cc:464] Registering TS with master...
I20260621 05:10:31.395923 6364 heartbeater.cc:511] Master 127.3.131.190:46269 requested a full tablet report, sending...
I20260621 05:10:31.396436 6039 ts_manager.cc:194] Registered new tserver with Master: 9cc82f280100477a81e260f6e1056525 (127.3.131.130:39577)
WARNING: no policy specified for kudu/127.3.131.131@KRBTEST.COM; defaulting to no policy
I20260621 05:10:31.396950 6039 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.131.130@KRBTEST.COM'} at 127.3.131.130:51013
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.131.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.131.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782018616648578-3598-0/krb5kdc/kudu_127.3.131.131.keytab.
Entry for principal kudu/127.3.131.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782018616648578-3598-0/krb5kdc/kudu_127.3.131.131.keytab.
I20260621 05:10:31.419539 3598 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWZOx2_/build/release/bin/kudu
/tmp/dist-test-taskWZOx2_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782018616648578-3598-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782018616648578-3598-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782018616648578-3598-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782018616648578-3598-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.131.131:0
--local_ip_for_outbound_sockets=127.3.131.131
--webserver_interface=127.3.131.131
--webserver_port=0
--tserver_master_addrs=127.3.131.190:46269
--keytab_file=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782018616648578-3598-0/krb5kdc/kudu_127.3.131.131.keytab
--principal=kudu/127.3.131.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.131.148:37207
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782018616648578-3598-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782018616648578-3598-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782018616648578-3598-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 05:10:31.507927 6369 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 05:10:31.508308 6369 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 05:10:31.508361 6369 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 05:10:31.509644 6369 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 05:10:31.509704 6369 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 05:10:31.509907 6369 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.131.131
I20260621 05:10:31.511847 6369 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.131.148:37207
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782018616648578-3598-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782018616648578-3598-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782018616648578-3598-0/krb5kdc/kudu_127.3.131.131.keytab
--principal=kudu/127.3.131.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.131.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782018616648578-3598-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.131.131
--webserver_port=0
--tserver_master_addrs=127.3.131.190:46269
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6369
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.131.131
--log_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782018616648578-3598-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 21 Jun 2026 04:16:16 UTC on ccycloud.quasar-xycaha.root.comops.site
build id 4457881
I20260621 05:10:31.512072 6369 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 05:10:31.512356 6369 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 05:10:31.518447 6375 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:10:31.518451 6374 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:10:31.518808 6377 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:10:31.518723 6369 server_base.cc:1034] running on GCE node
I20260621 05:10:31.519084 6369 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 05:10:31.519428 6369 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 05:10:31.520609 6369 hybrid_clock.cc:648] HybridClock initialized: now 1782018631520584 us; error 46 us; skew 500 ppm
Jun 21 05:10:31 dist-test-slave-vrq1 krb5kdc[5994](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782018631, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.131.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 05:10:31.523383 6369 init.cc:377] Logged in from keytab as kudu/127.3.131.131@KRBTEST.COM (short username kudu)
I20260621 05:10:31.523897 6369 webserver.cc:460] Webserver started at http://127.3.131.131:46007/ using document root <none> and password file <none>
I20260621 05:10:31.524158 6369 fs_manager.cc:359] Metadata directory not provided
I20260621 05:10:31.524210 6369 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 05:10:31.524395 6369 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 05:10:31.525480 6369 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782018616648578-3598-0/minicluster-data/ts-2/data/instance:
uuid: "0577edf689534ef688657c9a7ef36eef"
format_stamp: "Formatted at 2026-06-21 05:10:31 on dist-test-slave-vrq1"
server_key: "039b48a933f814b049571ccd7f8c89ea"
server_key_iv: "8a117cacc804ca6d2ab04aeaebb1f24b"
server_key_version: "encryptionkey@0"
I20260621 05:10:31.525854 6369 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782018616648578-3598-0/minicluster-data/ts-2/wal/instance:
uuid: "0577edf689534ef688657c9a7ef36eef"
format_stamp: "Formatted at 2026-06-21 05:10:31 on dist-test-slave-vrq1"
server_key: "039b48a933f814b049571ccd7f8c89ea"
server_key_iv: "8a117cacc804ca6d2ab04aeaebb1f24b"
server_key_version: "encryptionkey@0"
I20260621 05:10:31.527328 6369 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260621 05:10:31.528178 6384 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 05:10:31.528364 6369 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260621 05:10:31.528463 6369 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782018616648578-3598-0/minicluster-data/ts-2/data,/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782018616648578-3598-0/minicluster-data/ts-2/wal
uuid: "0577edf689534ef688657c9a7ef36eef"
format_stamp: "Formatted at 2026-06-21 05:10:31 on dist-test-slave-vrq1"
server_key: "039b48a933f814b049571ccd7f8c89ea"
server_key_iv: "8a117cacc804ca6d2ab04aeaebb1f24b"
server_key_version: "encryptionkey@0"
I20260621 05:10:31.528546 6369 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782018616648578-3598-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782018616648578-3598-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782018616648578-3598-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 05:10:31.538841 6369 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 05:10:31.541193 6369 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 05:10:31.541363 6369 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 05:10:31.541610 6369 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 05:10:31.541932 6369 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 05:10:31.541987 6369 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 05:10:31.542030 6369 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 05:10:31.542055 6369 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 05:10:31.548094 6369 rpc_server.cc:307] RPC server started. Bound to: 127.3.131.131:46023
I20260621 05:10:31.548192 6497 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.131.131:46023 every 8 connection(s)
I20260621 05:10:31.548512 6369 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782018616648578-3598-0/minicluster-data/ts-2/data/info.pb
I20260621 05:10:31.554957 3598 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWZOx2_/build/release/bin/kudu as pid 6369
I20260621 05:10:31.555068 3598 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782018616648578-3598-0/minicluster-data/ts-2/wal/instance
I20260621 05:10:31.555341 3598 external_mini_cluster.cc:1428] Setting key 29b1628319d23e9a637d36e755a6a3c0
Jun 21 05:10:31 dist-test-slave-vrq1 krb5kdc[5994](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782018631, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.131.131@KRBTEST.COM for kudu/127.3.131.190@KRBTEST.COM
I20260621 05:10:31.561107 6096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:10:31.549418 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.131.131:56545 (local address 127.3.131.190:46269)
0621 05:10:31.549553 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0621 05:10:31.549555 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:10:31.549855 (+ 300us) server_negotiation.cc:407] Connection header received
0621 05:10:31.550346 (+ 491us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:10:31.550347 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:10:31.550358 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:10:31.550416 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:10:31.551532 (+ 1116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:10:31.552044 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:10:31.553541 (+ 1497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:10:31.553757 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:10:31.553841 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:10:31.557222 (+ 3381us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:10:31.557223 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:10:31.557224 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:10:31.557239 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:10:31.558862 (+ 1623us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:10:31.559699 (+ 837us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:10:31.559701 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:10:31.559701 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:10:31.559781 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:10:31.560135 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:10:31.560136 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:10:31.560136 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:10:31.560283 (+ 147us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:10:31.560373 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0621 05:10:31.560711 (+ 338us) server_negotiation.cc:299] Negotiation successful
0621 05:10:31.560814 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":96}
I20260621 05:10:31.561335 6500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:10:31.549407 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.131.190:46269 (local address 127.3.131.131:56545)
0621 05:10:31.549680 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0621 05:10:31.549729 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0621 05:10:31.550248 (+ 519us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:10:31.550475 (+ 227us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:10:31.550478 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:10:31.551083 (+ 605us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:10:31.551417 (+ 334us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:10:31.551425 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:10:31.552151 (+ 726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:10:31.552152 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:10:31.553410 (+ 1258us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:10:31.553414 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:10:31.553848 (+ 434us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:10:31.553849 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:10:31.553994 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:10:31.554837 (+ 843us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 05:10:31.554848 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:10:31.557108 (+ 2260us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 05:10:31.559151 (+ 2043us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:10:31.559162 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:10:31.559166 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:10:31.559548 (+ 382us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:10:31.559892 (+ 344us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:10:31.559894 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:10:31.559894 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:10:31.560048 (+ 154us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:10:31.560360 (+ 312us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 05:10:31.560368 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 05:10:31.560550 (+ 182us) client_negotiation.cc:769] Sending connection context
0621 05:10:31.560671 (+ 121us) client_negotiation.cc:240] Negotiation successful
0621 05:10:31.561052 (+ 381us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":115,"threads_started":1}
I20260621 05:10:31.562050 6498 heartbeater.cc:347] Connected to a master server at 127.3.131.190:46269
I20260621 05:10:31.562155 6498 heartbeater.cc:464] Registering TS with master...
I20260621 05:10:31.562433 6498 heartbeater.cc:511] Master 127.3.131.190:46269 requested a full tablet report, sending...
I20260621 05:10:31.563133 6039 ts_manager.cc:194] Registered new tserver with Master: 0577edf689534ef688657c9a7ef36eef (127.3.131.131:46023)
I20260621 05:10:31.563719 6039 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.131.131@KRBTEST.COM'} at 127.3.131.131:56545
I20260621 05:10:31.568745 3598 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260621 05:10:31.582804 6096 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 05:10:31.581539 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33954 (local address 127.3.131.190:46269)
0621 05:10:31.581633 (+ 94us) server_negotiation.cc:206] Beginning negotiation
0621 05:10:31.581636 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0621 05:10:31.581646 (+ 10us) server_negotiation.cc:407] Connection header received
0621 05:10:31.581674 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:10:31.581675 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:10:31.581685 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:10:31.581758 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:10:31.582701 (+ 943us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33954: BlockingRecv error: recv got EOF from 127.0.0.1:33954 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
I20260621 05:10:31.584710 3598 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWZOx2_/build/release/bin/kudu with pid 6101
I20260621 05:10:31.589603 3598 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWZOx2_/build/release/bin/kudu with pid 6235
I20260621 05:10:31.594292 3598 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWZOx2_/build/release/bin/kudu with pid 6369
I20260621 05:10:31.599313 3598 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWZOx2_/build/release/bin/kudu with pid 6007
2026-06-21T05:10:31Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (1930 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1782018616648578-3598-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 21 05:10:31 dist-test-slave-vrq1 krb5kdc[6510](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 21 05:10:31 dist-test-slave-vrq1 krb5kdc[6510](info): set up 2 sockets
Jun 21 05:10:31 dist-test-slave-vrq1 krb5kdc[6510](info): commencing operation
krb5kdc: starting...
W20260621 05:10:32.679591 3598 mini_kdc.cc:121] Time spent starting KDC: real 1.059s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1782018616648578-3598-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1782018616648578-3598-0/krb5kdc/test-user.keytab.
Jun 21 05:10:32 dist-test-slave-vrq1 krb5kdc[6510](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782018632, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-21T05:10:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-21T05:10:32Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.131.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.131.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.131.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1782018616648578-3598-0/krb5kdc/kudu_127.3.131.190.keytab.
Entry for principal kudu/127.3.131.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1782018616648578-3598-0/krb5kdc/kudu_127.3.131.190.keytab.
I20260621 05:10:32.811497 3598 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWZOx2_/build/release/bin/kudu
/tmp/dist-test-taskWZOx2_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1782018616648578-3598-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1782018616648578-3598-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1782018616648578-3598-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1782018616648578-3598-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.131.190:41187
--webserver_interface=127.3.131.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1782018616648578-3598-0/krb5kdc/kudu_127.3.131.190.keytab
--principal=kudu/127.3.131.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.131.148:36413
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.131.190:41187
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1782018616648578-3598-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1782018616648578-3598-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1782018616648578-3598-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 05:10:32.893713 6523 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 05:10:32.893924 6523 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 05:10:32.893947 6523 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 05:10:32.895244 6523 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 05:10:32.895289 6523 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 05:10:32.895306 6523 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 05:10:32.895319 6523 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 05:10:32.895331 6523 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 05:10:32.896917 6523 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.131.148:36413
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1782018616648578-3598-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1782018616648578-3598-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.131.190:41187
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1782018616648578-3598-0/krb5kdc/kudu_127.3.131.190.keytab
--principal=kudu/127.3.131.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.131.190:41187
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1782018616648578-3598-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.131.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6523
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1782018616648578-3598-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 21 Jun 2026 04:16:16 UTC on ccycloud.quasar-xycaha.root.comops.site
build id 4457881
I20260621 05:10:32.897187 6523 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 05:10:32.897429 6523 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 05:10:32.902563 6529 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:10:32.902630 6528 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:10:32.902565 6531 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:10:32.902653 6523 server_base.cc:1034] running on GCE node
I20260621 05:10:32.903040 6523 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 05:10:32.903374 6523 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 05:10:32.904570 6523 hybrid_clock.cc:648] HybridClock initialized: now 1782018632904547 us; error 68 us; skew 500 ppm
Jun 21 05:10:32 dist-test-slave-vrq1 krb5kdc[6510](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782018632, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.131.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 05:10:32.911612 6523 init.cc:377] Logged in from keytab as kudu/127.3.131.190@KRBTEST.COM (short username kudu)
I20260621 05:10:32.912074 6523 webserver.cc:460] Webserver started at http://127.3.131.190:44979/ using document root <none> and password file <none>
I20260621 05:10:32.912286 6523 fs_manager.cc:359] Metadata directory not provided
I20260621 05:10:32.912358 6523 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 05:10:32.912458 6523 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 05:10:32.913550 6523 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1782018616648578-3598-0/minicluster-data/master-0/data/instance:
uuid: "4a6fde0c6bbc44128fd784778a4efa11"
format_stamp: "Formatted at 2026-06-21 05:10:32 on dist-test-slave-vrq1"
server_key: "fae01b97582a05addba294f6cd6110b5"
server_key_iv: "d02636375e50087468146a985d550961"
server_key_version: "encryptionkey@0"
I20260621 05:10:32.913887 6523 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1782018616648578-3598-0/minicluster-data/master-0/wal/instance:
uuid: "4a6fde0c6bbc44128fd784778a4efa11"
format_stamp: "Formatted at 2026-06-21 05:10:32 on dist-test-slave-vrq1"
server_key: "fae01b97582a05addba294f6cd6110b5"
server_key_iv: "d02636375e50087468146a985d550961"
server_key_version: "encryptionkey@0"
I20260621 05:10:32.915292 6523 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260621 05:10:32.916118 6538 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 05:10:32.916333 6523 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260621 05:10:32.916426 6523 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1782018616648578-3598-0/minicluster-data/master-0/data,/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1782018616648578-3598-0/minicluster-data/master-0/wal
uuid: "4a6fde0c6bbc44128fd784778a4efa11"
format_stamp: "Formatted at 2026-06-21 05:10:32 on dist-test-slave-vrq1"
server_key: "fae01b97582a05addba294f6cd6110b5"
server_key_iv: "d02636375e50087468146a985d550961"
server_key_version: "encryptionkey@0"
I20260621 05:10:32.916528 6523 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1782018616648578-3598-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1782018616648578-3598-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1782018616648578-3598-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 05:10:32.933586 6523 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 05:10:32.936034 6523 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 05:10:32.936185 6523 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 05:10:32.940418 6523 rpc_server.cc:307] RPC server started. Bound to: 127.3.131.190:41187
I20260621 05:10:32.940506 6590 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.131.190:41187 every 8 connection(s)
I20260621 05:10:32.941007 6523 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1782018616648578-3598-0/minicluster-data/master-0/data/info.pb
I20260621 05:10:32.942116 6591 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 05:10:32.944463 6591 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4a6fde0c6bbc44128fd784778a4efa11: Bootstrap starting.
I20260621 05:10:32.945034 6591 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4a6fde0c6bbc44128fd784778a4efa11: Neither blocks nor log segments found. Creating new log.
I20260621 05:10:32.945305 6591 log.cc:824] T 00000000000000000000000000000000 P 4a6fde0c6bbc44128fd784778a4efa11: Log is configured to *not* fsync() on all Append() calls
I20260621 05:10:32.946118 6591 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4a6fde0c6bbc44128fd784778a4efa11: No bootstrap required, opened a new log
I20260621 05:10:32.946930 3598 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWZOx2_/build/release/bin/kudu as pid 6523
I20260621 05:10:32.947031 3598 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1782018616648578-3598-0/minicluster-data/master-0/wal/instance
I20260621 05:10:32.947227 3598 external_mini_cluster.cc:1428] Setting key d0ca31bd72002f87f188bedce74b3a9f
I20260621 05:10:32.948832 6591 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4a6fde0c6bbc44128fd784778a4efa11 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a6fde0c6bbc44128fd784778a4efa11" member_type: VOTER last_known_addr { host: "127.3.131.190" port: 41187 } }
I20260621 05:10:32.948978 6591 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4a6fde0c6bbc44128fd784778a4efa11 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 05:10:32.949000 6591 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4a6fde0c6bbc44128fd784778a4efa11 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4a6fde0c6bbc44128fd784778a4efa11, State: Initialized, Role: FOLLOWER
I20260621 05:10:32.949147 6591 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4a6fde0c6bbc44128fd784778a4efa11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a6fde0c6bbc44128fd784778a4efa11" member_type: VOTER last_known_addr { host: "127.3.131.190" port: 41187 } }
I20260621 05:10:32.949234 6591 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4a6fde0c6bbc44128fd784778a4efa11 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260621 05:10:32.949256 6591 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4a6fde0c6bbc44128fd784778a4efa11 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260621 05:10:32.949326 6591 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4a6fde0c6bbc44128fd784778a4efa11 [term 0 FOLLOWER]: Advancing to term 1
I20260621 05:10:32.950177 6591 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4a6fde0c6bbc44128fd784778a4efa11 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a6fde0c6bbc44128fd784778a4efa11" member_type: VOTER last_known_addr { host: "127.3.131.190" port: 41187 } }
I20260621 05:10:32.950296 6591 leader_election.cc:302] T 00000000000000000000000000000000 P 4a6fde0c6bbc44128fd784778a4efa11 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4a6fde0c6bbc44128fd784778a4efa11; no voters:
I20260621 05:10:32.950517 6591 leader_election.cc:288] T 00000000000000000000000000000000 P 4a6fde0c6bbc44128fd784778a4efa11 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260621 05:10:32.950555 6596 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4a6fde0c6bbc44128fd784778a4efa11 [term 1 FOLLOWER]: Leader election won for term 1
I20260621 05:10:32.950660 6596 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4a6fde0c6bbc44128fd784778a4efa11 [term 1 LEADER]: Becoming Leader. State: Replica: 4a6fde0c6bbc44128fd784778a4efa11, State: Running, Role: LEADER
I20260621 05:10:32.950747 6596 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4a6fde0c6bbc44128fd784778a4efa11 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a6fde0c6bbc44128fd784778a4efa11" member_type: VOTER last_known_addr { host: "127.3.131.190" port: 41187 } }
I20260621 05:10:32.950814 6591 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4a6fde0c6bbc44128fd784778a4efa11 [sys.catalog]: configured and running, proceeding with master startup.
I20260621 05:10:32.951109 6596 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4a6fde0c6bbc44128fd784778a4efa11 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4a6fde0c6bbc44128fd784778a4efa11" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a6fde0c6bbc44128fd784778a4efa11" member_type: VOTER last_known_addr { host: "127.3.131.190" port: 41187 } } }
I20260621 05:10:32.951290 6596 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4a6fde0c6bbc44128fd784778a4efa11 [sys.catalog]: This master's current role is: LEADER
I20260621 05:10:32.951486 6596 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4a6fde0c6bbc44128fd784778a4efa11 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4a6fde0c6bbc44128fd784778a4efa11. Latest consensus state: current_term: 1 leader_uuid: "4a6fde0c6bbc44128fd784778a4efa11" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a6fde0c6bbc44128fd784778a4efa11" member_type: VOTER last_known_addr { host: "127.3.131.190" port: 41187 } } }
I20260621 05:10:32.951578 6596 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4a6fde0c6bbc44128fd784778a4efa11 [sys.catalog]: This master's current role is: LEADER
I20260621 05:10:32.951927 6603 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260621 05:10:32.952584 6603 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260621 05:10:32.954681 6603 catalog_manager.cc:1347] Generated new cluster ID: 011af20506b44ec49c0b5e459e8f1395
I20260621 05:10:32.954782 6603 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Jun 21 05:10:32 dist-test-slave-vrq1 krb5kdc[6510](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782018632, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.131.190@KRBTEST.COM
I20260621 05:10:32.959805 6595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:10:32.947839 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52306 (local address 127.3.131.190:41187)
0621 05:10:32.948058 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0621 05:10:32.948066 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0621 05:10:32.948093 (+ 27us) server_negotiation.cc:407] Connection header received
0621 05:10:32.949048 (+ 955us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:10:32.949059 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:10:32.949198 (+ 139us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:10:32.949372 (+ 174us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:10:32.950238 (+ 866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:10:32.951759 (+ 1521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:10:32.952919 (+ 1160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:10:32.953533 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:10:32.953655 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:10:32.956398 (+ 2743us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:10:32.956402 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:10:32.956403 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:10:32.956415 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:10:32.958170 (+ 1755us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:10:32.958584 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:10:32.958592 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:10:32.958592 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:10:32.958693 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:10:32.958908 (+ 215us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:10:32.958909 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:10:32.958909 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:10:32.959165 (+ 256us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:10:32.959408 (+ 243us) server_negotiation.cc:1035] Waiting for connection context
0621 05:10:32.959480 (+ 72us) server_negotiation.cc:299] Negotiation successful
0621 05:10:32.959628 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":156,"thread_start_us":111,"threads_started":1}
I20260621 05:10:32.960726 3598 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260621 05:10:32.972966 6603 catalog_manager.cc:1370] Generated new certificate authority record
I20260621 05:10:32.973557 6603 catalog_manager.cc:1504] Loading token signing keys...
I20260621 05:10:32.979722 6603 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4a6fde0c6bbc44128fd784778a4efa11: Generated new TSK 0
I20260621 05:10:32.979949 6603 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260621 05:10:32.983587 6595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:10:32.978489 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52314 (local address 127.3.131.190:41187)
0621 05:10:32.978537 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0621 05:10:32.978540 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0621 05:10:32.978682 (+ 142us) server_negotiation.cc:407] Connection header received
0621 05:10:32.978940 (+ 258us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:10:32.978942 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:10:32.978956 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:10:32.978993 (+ 37us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:10:32.979644 (+ 651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:10:32.980534 (+ 890us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:10:32.981857 (+ 1323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:10:32.982095 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:10:32.982159 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:10:32.982974 (+ 815us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:10:32.982975 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:10:32.982976 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 05:10:32.983287 (+ 311us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:10:32.983308 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 05:10:32.983315 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:10:32.983369 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0621 05:10:32.983463 (+ 94us) server_negotiation.cc:299] Negotiation successful
0621 05:10:32.983487 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260621 05:10:32.985939 3598 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWZOx2_/build/release/bin/kudu with pid 6523
2026-06-21T05:10:32Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (1383 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782018616648578-3598-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 21 05:10:33 dist-test-slave-vrq1 krb5kdc[6624](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 21 05:10:33 dist-test-slave-vrq1 krb5kdc[6624](info): set up 2 sockets
Jun 21 05:10:33 dist-test-slave-vrq1 krb5kdc[6624](info): commencing operation
krb5kdc: starting...
W20260621 05:10:34.087544 3598 mini_kdc.cc:121] Time spent starting KDC: real 1.083s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782018616648578-3598-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782018616648578-3598-0/krb5kdc/test-user.keytab.
Jun 21 05:10:34 dist-test-slave-vrq1 krb5kdc[6624](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782018634, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-21T05:10:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-21T05:10:34Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.131.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.131.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.131.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782018616648578-3598-0/krb5kdc/kudu_127.3.131.190.keytab.
Entry for principal kudu/127.3.131.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782018616648578-3598-0/krb5kdc/kudu_127.3.131.190.keytab.
I20260621 05:10:34.211480 3598 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWZOx2_/build/release/bin/kudu
/tmp/dist-test-taskWZOx2_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782018616648578-3598-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782018616648578-3598-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782018616648578-3598-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782018616648578-3598-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.131.190:42441
--webserver_interface=127.3.131.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782018616648578-3598-0/krb5kdc/kudu_127.3.131.190.keytab
--principal=kudu/127.3.131.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.131.148:45413
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.131.190:42441
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782018616648578-3598-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782018616648578-3598-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782018616648578-3598-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 05:10:34.293005 6637 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 05:10:34.293280 6637 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 05:10:34.293319 6637 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 05:10:34.294636 6637 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 05:10:34.294735 6637 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 05:10:34.294761 6637 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 05:10:34.294791 6637 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 05:10:34.294823 6637 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 05:10:34.296348 6637 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.131.148:45413
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782018616648578-3598-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782018616648578-3598-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.131.190:42441
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782018616648578-3598-0/krb5kdc/kudu_127.3.131.190.keytab
--principal=kudu/127.3.131.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.131.190:42441
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782018616648578-3598-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.131.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6637
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782018616648578-3598-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 21 Jun 2026 04:16:16 UTC on ccycloud.quasar-xycaha.root.comops.site
build id 4457881
I20260621 05:10:34.296623 6637 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 05:10:34.296854 6637 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 05:10:34.302372 6643 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:10:34.302390 6642 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:10:34.302428 6645 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:10:34.302685 6637 server_base.cc:1034] running on GCE node
I20260621 05:10:34.303220 6637 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 05:10:34.303596 6637 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 05:10:34.304795 6637 hybrid_clock.cc:648] HybridClock initialized: now 1782018634304761 us; error 54 us; skew 500 ppm
Jun 21 05:10:34 dist-test-slave-vrq1 krb5kdc[6624](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782018634, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.131.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 05:10:34.307797 6637 init.cc:377] Logged in from keytab as kudu/127.3.131.190@KRBTEST.COM (short username kudu)
I20260621 05:10:34.308266 6637 webserver.cc:460] Webserver started at http://127.3.131.190:38071/ using document root <none> and password file <none>
I20260621 05:10:34.308506 6637 fs_manager.cc:359] Metadata directory not provided
I20260621 05:10:34.308611 6637 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 05:10:34.308771 6637 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 05:10:34.309938 6637 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782018616648578-3598-0/minicluster-data/master-0/data/instance:
uuid: "fc8a06302b2246948a65143417c144b1"
format_stamp: "Formatted at 2026-06-21 05:10:34 on dist-test-slave-vrq1"
server_key: "1c5133701ae169ce6ef0c7cfee119352"
server_key_iv: "f5d8b0e7d09e0570c5573032da66a1d9"
server_key_version: "encryptionkey@0"
I20260621 05:10:34.310328 6637 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782018616648578-3598-0/minicluster-data/master-0/wal/instance:
uuid: "fc8a06302b2246948a65143417c144b1"
format_stamp: "Formatted at 2026-06-21 05:10:34 on dist-test-slave-vrq1"
server_key: "1c5133701ae169ce6ef0c7cfee119352"
server_key_iv: "f5d8b0e7d09e0570c5573032da66a1d9"
server_key_version: "encryptionkey@0"
I20260621 05:10:34.312064 6637 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.003s
I20260621 05:10:34.312951 6652 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 05:10:34.313151 6637 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260621 05:10:34.313269 6637 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782018616648578-3598-0/minicluster-data/master-0/data,/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782018616648578-3598-0/minicluster-data/master-0/wal
uuid: "fc8a06302b2246948a65143417c144b1"
format_stamp: "Formatted at 2026-06-21 05:10:34 on dist-test-slave-vrq1"
server_key: "1c5133701ae169ce6ef0c7cfee119352"
server_key_iv: "f5d8b0e7d09e0570c5573032da66a1d9"
server_key_version: "encryptionkey@0"
I20260621 05:10:34.313369 6637 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782018616648578-3598-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782018616648578-3598-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782018616648578-3598-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 05:10:34.354409 6637 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 05:10:34.356966 6637 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 05:10:34.357164 6637 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 05:10:34.361423 6637 rpc_server.cc:307] RPC server started. Bound to: 127.3.131.190:42441
I20260621 05:10:34.361452 6704 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.131.190:42441 every 8 connection(s)
I20260621 05:10:34.361830 6637 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782018616648578-3598-0/minicluster-data/master-0/data/info.pb
I20260621 05:10:34.362892 6705 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 05:10:34.365238 6705 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fc8a06302b2246948a65143417c144b1: Bootstrap starting.
I20260621 05:10:34.365803 6705 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fc8a06302b2246948a65143417c144b1: Neither blocks nor log segments found. Creating new log.
I20260621 05:10:34.366039 6705 log.cc:824] T 00000000000000000000000000000000 P fc8a06302b2246948a65143417c144b1: Log is configured to *not* fsync() on all Append() calls
I20260621 05:10:34.366730 6705 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fc8a06302b2246948a65143417c144b1: No bootstrap required, opened a new log
I20260621 05:10:34.367313 3598 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWZOx2_/build/release/bin/kudu as pid 6637
I20260621 05:10:34.367437 3598 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782018616648578-3598-0/minicluster-data/master-0/wal/instance
I20260621 05:10:34.367626 3598 external_mini_cluster.cc:1428] Setting key 367b195a30cb43e444daede5c43bb978
I20260621 05:10:34.368965 6705 raft_consensus.cc:348] T 00000000000000000000000000000000 P fc8a06302b2246948a65143417c144b1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc8a06302b2246948a65143417c144b1" member_type: VOTER last_known_addr { host: "127.3.131.190" port: 42441 } }
I20260621 05:10:34.369160 6705 raft_consensus.cc:374] T 00000000000000000000000000000000 P fc8a06302b2246948a65143417c144b1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 05:10:34.369210 6705 raft_consensus.cc:729] T 00000000000000000000000000000000 P fc8a06302b2246948a65143417c144b1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fc8a06302b2246948a65143417c144b1, State: Initialized, Role: FOLLOWER
I20260621 05:10:34.369376 6705 consensus_queue.cc:260] T 00000000000000000000000000000000 P fc8a06302b2246948a65143417c144b1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc8a06302b2246948a65143417c144b1" member_type: VOTER last_known_addr { host: "127.3.131.190" port: 42441 } }
I20260621 05:10:34.369475 6705 raft_consensus.cc:388] T 00000000000000000000000000000000 P fc8a06302b2246948a65143417c144b1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260621 05:10:34.369519 6705 raft_consensus.cc:482] T 00000000000000000000000000000000 P fc8a06302b2246948a65143417c144b1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260621 05:10:34.369596 6705 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fc8a06302b2246948a65143417c144b1 [term 0 FOLLOWER]: Advancing to term 1
I20260621 05:10:34.370223 6705 raft_consensus.cc:504] T 00000000000000000000000000000000 P fc8a06302b2246948a65143417c144b1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc8a06302b2246948a65143417c144b1" member_type: VOTER last_known_addr { host: "127.3.131.190" port: 42441 } }
I20260621 05:10:34.370347 6705 leader_election.cc:302] T 00000000000000000000000000000000 P fc8a06302b2246948a65143417c144b1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fc8a06302b2246948a65143417c144b1; no voters:
I20260621 05:10:34.370584 6705 leader_election.cc:288] T 00000000000000000000000000000000 P fc8a06302b2246948a65143417c144b1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260621 05:10:34.370609 6710 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fc8a06302b2246948a65143417c144b1 [term 1 FOLLOWER]: Leader election won for term 1
I20260621 05:10:34.370738 6710 raft_consensus.cc:686] T 00000000000000000000000000000000 P fc8a06302b2246948a65143417c144b1 [term 1 LEADER]: Becoming Leader. State: Replica: fc8a06302b2246948a65143417c144b1, State: Running, Role: LEADER
I20260621 05:10:34.370877 6705 sys_catalog.cc:564] T 00000000000000000000000000000000 P fc8a06302b2246948a65143417c144b1 [sys.catalog]: configured and running, proceeding with master startup.
I20260621 05:10:34.370860 6710 consensus_queue.cc:237] T 00000000000000000000000000000000 P fc8a06302b2246948a65143417c144b1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc8a06302b2246948a65143417c144b1" member_type: VOTER last_known_addr { host: "127.3.131.190" port: 42441 } }
I20260621 05:10:34.371234 6710 sys_catalog.cc:455] T 00000000000000000000000000000000 P fc8a06302b2246948a65143417c144b1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fc8a06302b2246948a65143417c144b1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc8a06302b2246948a65143417c144b1" member_type: VOTER last_known_addr { host: "127.3.131.190" port: 42441 } } }
I20260621 05:10:34.371366 6710 sys_catalog.cc:458] T 00000000000000000000000000000000 P fc8a06302b2246948a65143417c144b1 [sys.catalog]: This master's current role is: LEADER
I20260621 05:10:34.371472 6712 sys_catalog.cc:455] T 00000000000000000000000000000000 P fc8a06302b2246948a65143417c144b1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader fc8a06302b2246948a65143417c144b1. Latest consensus state: current_term: 1 leader_uuid: "fc8a06302b2246948a65143417c144b1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc8a06302b2246948a65143417c144b1" member_type: VOTER last_known_addr { host: "127.3.131.190" port: 42441 } } }
I20260621 05:10:34.371618 6712 sys_catalog.cc:458] T 00000000000000000000000000000000 P fc8a06302b2246948a65143417c144b1 [sys.catalog]: This master's current role is: LEADER
I20260621 05:10:34.371630 6717 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260621 05:10:34.372241 6717 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260621 05:10:34.374290 6717 catalog_manager.cc:1347] Generated new cluster ID: 4b78217c52124934982f8e84563307be
I20260621 05:10:34.374346 6717 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Jun 21 05:10:34 dist-test-slave-vrq1 krb5kdc[6624](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782018634, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.131.190@KRBTEST.COM
I20260621 05:10:34.379454 6709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:10:34.368188 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48140 (local address 127.3.131.190:42441)
0621 05:10:34.368364 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0621 05:10:34.368374 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0621 05:10:34.368412 (+ 38us) server_negotiation.cc:407] Connection header received
0621 05:10:34.369092 (+ 680us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:10:34.369100 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:10:34.369157 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:10:34.369353 (+ 196us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:10:34.370057 (+ 704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:10:34.371325 (+ 1268us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:10:34.372415 (+ 1090us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:10:34.373077 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:10:34.373197 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:10:34.376030 (+ 2833us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:10:34.376035 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:10:34.376035 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:10:34.376049 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:10:34.377814 (+ 1765us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:10:34.378224 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:10:34.378228 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:10:34.378228 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:10:34.378328 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:10:34.378544 (+ 216us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:10:34.378545 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:10:34.378545 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:10:34.378801 (+ 256us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:10:34.378993 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0621 05:10:34.379072 (+ 79us) server_negotiation.cc:299] Negotiation successful
0621 05:10:34.379266 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":119,"thread_start_us":62,"threads_started":1}
I20260621 05:10:34.406788 6717 catalog_manager.cc:1370] Generated new certificate authority record
I20260621 05:10:34.407582 6717 catalog_manager.cc:1504] Loading token signing keys...
I20260621 05:10:34.414505 6717 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fc8a06302b2246948a65143417c144b1: Generated new TSK 0
I20260621 05:10:34.414752 6717 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.3.131.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.131.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.131.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782018616648578-3598-0/krb5kdc/kudu_127.3.131.129.keytab.
Entry for principal kudu/127.3.131.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782018616648578-3598-0/krb5kdc/kudu_127.3.131.129.keytab.
I20260621 05:10:34.447916 3598 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWZOx2_/build/release/bin/kudu
/tmp/dist-test-taskWZOx2_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782018616648578-3598-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782018616648578-3598-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782018616648578-3598-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782018616648578-3598-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.131.129:0
--local_ip_for_outbound_sockets=127.3.131.129
--webserver_interface=127.3.131.129
--webserver_port=0
--tserver_master_addrs=127.3.131.190:42441
--keytab_file=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782018616648578-3598-0/krb5kdc/kudu_127.3.131.129.keytab
--principal=kudu/127.3.131.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.131.148:45413
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782018616648578-3598-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782018616648578-3598-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782018616648578-3598-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 05:10:34.545883 6731 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 05:10:34.546144 6731 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 05:10:34.546172 6731 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 05:10:34.547420 6731 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 05:10:34.547465 6731 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 05:10:34.547493 6731 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.131.129
I20260621 05:10:34.549018 6731 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.131.148:45413
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782018616648578-3598-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782018616648578-3598-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782018616648578-3598-0/krb5kdc/kudu_127.3.131.129.keytab
--principal=kudu/127.3.131.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.131.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782018616648578-3598-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.131.129
--webserver_port=0
--tserver_master_addrs=127.3.131.190:42441
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6731
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.131.129
--log_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782018616648578-3598-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 21 Jun 2026 04:16:16 UTC on ccycloud.quasar-xycaha.root.comops.site
build id 4457881
I20260621 05:10:34.549244 6731 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 05:10:34.549437 6731 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260621 05:10:34.556336 6731 server_base.cc:1034] running on GCE node
W20260621 05:10:34.556296 6737 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:10:34.556313 6739 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:10:34.556313 6736 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:10:34.556979 6731 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 05:10:34.557461 6731 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 05:10:34.558681 6731 hybrid_clock.cc:648] HybridClock initialized: now 1782018634558635 us; error 65 us; skew 500 ppm
Jun 21 05:10:34 dist-test-slave-vrq1 krb5kdc[6624](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782018634, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.131.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 05:10:34.562372 6731 init.cc:377] Logged in from keytab as kudu/127.3.131.129@KRBTEST.COM (short username kudu)
I20260621 05:10:34.562973 6731 webserver.cc:460] Webserver started at http://127.3.131.129:42115/ using document root <none> and password file <none>
I20260621 05:10:34.563300 6731 fs_manager.cc:359] Metadata directory not provided
I20260621 05:10:34.563377 6731 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 05:10:34.563546 6731 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 05:10:34.564769 6731 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782018616648578-3598-0/minicluster-data/ts-0/data/instance:
uuid: "0773703171b741069ca162fffa17fc83"
format_stamp: "Formatted at 2026-06-21 05:10:34 on dist-test-slave-vrq1"
server_key: "046491c98c98a4df4fed308f3e1c4106"
server_key_iv: "09149d47feafe13d4c91234ec566d542"
server_key_version: "encryptionkey@0"
I20260621 05:10:34.565162 6731 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782018616648578-3598-0/minicluster-data/ts-0/wal/instance:
uuid: "0773703171b741069ca162fffa17fc83"
format_stamp: "Formatted at 2026-06-21 05:10:34 on dist-test-slave-vrq1"
server_key: "046491c98c98a4df4fed308f3e1c4106"
server_key_iv: "09149d47feafe13d4c91234ec566d542"
server_key_version: "encryptionkey@0"
I20260621 05:10:34.566651 6731 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260621 05:10:34.567672 6746 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 05:10:34.567910 6731 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260621 05:10:34.568023 6731 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782018616648578-3598-0/minicluster-data/ts-0/data,/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782018616648578-3598-0/minicluster-data/ts-0/wal
uuid: "0773703171b741069ca162fffa17fc83"
format_stamp: "Formatted at 2026-06-21 05:10:34 on dist-test-slave-vrq1"
server_key: "046491c98c98a4df4fed308f3e1c4106"
server_key_iv: "09149d47feafe13d4c91234ec566d542"
server_key_version: "encryptionkey@0"
I20260621 05:10:34.568164 6731 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782018616648578-3598-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782018616648578-3598-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782018616648578-3598-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 05:10:34.584336 6731 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 05:10:34.587803 6731 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 05:10:34.587996 6731 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 05:10:34.588342 6731 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 05:10:34.588814 6731 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 05:10:34.588867 6731 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 05:10:34.588936 6731 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 05:10:34.588982 6731 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 05:10:34.597466 6731 rpc_server.cc:307] RPC server started. Bound to: 127.3.131.129:42263
I20260621 05:10:34.597575 6859 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.131.129:42263 every 8 connection(s)
I20260621 05:10:34.598048 6731 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782018616648578-3598-0/minicluster-data/ts-0/data/info.pb
I20260621 05:10:34.604260 3598 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWZOx2_/build/release/bin/kudu as pid 6731
I20260621 05:10:34.604396 3598 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782018616648578-3598-0/minicluster-data/ts-0/wal/instance
I20260621 05:10:34.604635 3598 external_mini_cluster.cc:1428] Setting key 2e4ebbe3a6b28ef565c71aa514366b2c
Jun 21 05:10:34 dist-test-slave-vrq1 krb5kdc[6624](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782018634, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.131.129@KRBTEST.COM for kudu/127.3.131.190@KRBTEST.COM
I20260621 05:10:34.614275 6709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:10:34.599212 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.131.129:52833 (local address 127.3.131.190:42441)
0621 05:10:34.599283 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0621 05:10:34.599286 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0621 05:10:34.599610 (+ 324us) server_negotiation.cc:407] Connection header received
0621 05:10:34.600692 (+ 1082us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:10:34.600695 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:10:34.600714 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:10:34.600804 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:10:34.602073 (+ 1269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:10:34.602965 (+ 892us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:10:34.604373 (+ 1408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:10:34.604709 (+ 336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:10:34.604776 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:10:34.609675 (+ 4899us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:10:34.609678 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:10:34.609678 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:10:34.609688 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:10:34.611685 (+ 1997us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:10:34.612432 (+ 747us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:10:34.612434 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:10:34.612435 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:10:34.612515 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:10:34.612870 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:10:34.612871 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:10:34.612871 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:10:34.613037 (+ 166us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:10:34.613224 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0621 05:10:34.613739 (+ 515us) server_negotiation.cc:299] Negotiation successful
0621 05:10:34.614060 (+ 321us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260621 05:10:34.615340 6862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:10:34.599159 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.131.190:42441 (local address 127.3.131.129:52833)
0621 05:10:34.599383 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0621 05:10:34.599445 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0621 05:10:34.600493 (+ 1048us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:10:34.600896 (+ 403us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:10:34.600907 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:10:34.601726 (+ 819us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:10:34.601951 (+ 225us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:10:34.601956 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:10:34.603098 (+ 1142us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:10:34.603099 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:10:34.604266 (+ 1167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:10:34.604270 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:10:34.604971 (+ 701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:10:34.604973 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:10:34.605202 (+ 229us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:10:34.607523 (+ 2321us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 05:10:34.607538 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:10:34.609531 (+ 1993us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 05:10:34.611831 (+ 2300us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:10:34.611846 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:10:34.611850 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:10:34.612308 (+ 458us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:10:34.612595 (+ 287us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:10:34.612596 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:10:34.612597 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:10:34.612782 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:10:34.613281 (+ 499us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 05:10:34.613291 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 05:10:34.613534 (+ 243us) client_negotiation.cc:769] Sending connection context
0621 05:10:34.614713 (+ 1179us) client_negotiation.cc:240] Negotiation successful
0621 05:10:34.614971 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":89,"threads_started":1}
WARNING: no policy specified for kudu/127.3.131.130@KRBTEST.COM; defaulting to no policy
I20260621 05:10:34.616411 6860 heartbeater.cc:347] Connected to a master server at 127.3.131.190:42441
I20260621 05:10:34.616555 6860 heartbeater.cc:464] Registering TS with master...
I20260621 05:10:34.616796 6860 heartbeater.cc:511] Master 127.3.131.190:42441 requested a full tablet report, sending...
I20260621 05:10:34.617568 6669 ts_manager.cc:194] Registered new tserver with Master: 0773703171b741069ca162fffa17fc83 (127.3.131.129:42263)
I20260621 05:10:34.618655 6669 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.131.129@KRBTEST.COM'} at 127.3.131.129:52833
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.131.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.131.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782018616648578-3598-0/krb5kdc/kudu_127.3.131.130.keytab.
Entry for principal kudu/127.3.131.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782018616648578-3598-0/krb5kdc/kudu_127.3.131.130.keytab.
I20260621 05:10:34.643419 3598 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWZOx2_/build/release/bin/kudu
/tmp/dist-test-taskWZOx2_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782018616648578-3598-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782018616648578-3598-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782018616648578-3598-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782018616648578-3598-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.131.130:0
--local_ip_for_outbound_sockets=127.3.131.130
--webserver_interface=127.3.131.130
--webserver_port=0
--tserver_master_addrs=127.3.131.190:42441
--keytab_file=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782018616648578-3598-0/krb5kdc/kudu_127.3.131.130.keytab
--principal=kudu/127.3.131.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.131.148:45413
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782018616648578-3598-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782018616648578-3598-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782018616648578-3598-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 05:10:34.730139 6865 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 05:10:34.730415 6865 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 05:10:34.730459 6865 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 05:10:34.731802 6865 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 05:10:34.731889 6865 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 05:10:34.731940 6865 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.131.130
I20260621 05:10:34.733619 6865 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.131.148:45413
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782018616648578-3598-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782018616648578-3598-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782018616648578-3598-0/krb5kdc/kudu_127.3.131.130.keytab
--principal=kudu/127.3.131.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.131.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782018616648578-3598-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.131.130
--webserver_port=0
--tserver_master_addrs=127.3.131.190:42441
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6865
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.131.130
--log_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782018616648578-3598-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 21 Jun 2026 04:16:16 UTC on ccycloud.quasar-xycaha.root.comops.site
build id 4457881
I20260621 05:10:34.733927 6865 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 05:10:34.734201 6865 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 05:10:34.739802 6873 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:10:34.739802 6871 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:10:34.740048 6865 server_base.cc:1034] running on GCE node
W20260621 05:10:34.739830 6870 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:10:34.740440 6865 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 05:10:34.740660 6865 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 05:10:34.741799 6865 hybrid_clock.cc:648] HybridClock initialized: now 1782018634741774 us; error 40 us; skew 500 ppm
Jun 21 05:10:34 dist-test-slave-vrq1 krb5kdc[6624](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782018634, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.131.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 05:10:34.744638 6865 init.cc:377] Logged in from keytab as kudu/127.3.131.130@KRBTEST.COM (short username kudu)
I20260621 05:10:34.745122 6865 webserver.cc:460] Webserver started at http://127.3.131.130:35131/ using document root <none> and password file <none>
I20260621 05:10:34.745383 6865 fs_manager.cc:359] Metadata directory not provided
I20260621 05:10:34.745429 6865 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 05:10:34.745579 6865 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 05:10:34.746694 6865 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782018616648578-3598-0/minicluster-data/ts-1/data/instance:
uuid: "18455b1378024adf9a4f5092a093eb54"
format_stamp: "Formatted at 2026-06-21 05:10:34 on dist-test-slave-vrq1"
server_key: "44f60dee9efd6481d582745e7e5fc467"
server_key_iv: "c90f5450e9b7d97c96b95922aa991e3e"
server_key_version: "encryptionkey@0"
I20260621 05:10:34.747049 6865 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782018616648578-3598-0/minicluster-data/ts-1/wal/instance:
uuid: "18455b1378024adf9a4f5092a093eb54"
format_stamp: "Formatted at 2026-06-21 05:10:34 on dist-test-slave-vrq1"
server_key: "44f60dee9efd6481d582745e7e5fc467"
server_key_iv: "c90f5450e9b7d97c96b95922aa991e3e"
server_key_version: "encryptionkey@0"
I20260621 05:10:34.748438 6865 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260621 05:10:34.749279 6880 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 05:10:34.749490 6865 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260621 05:10:34.749598 6865 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782018616648578-3598-0/minicluster-data/ts-1/data,/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782018616648578-3598-0/minicluster-data/ts-1/wal
uuid: "18455b1378024adf9a4f5092a093eb54"
format_stamp: "Formatted at 2026-06-21 05:10:34 on dist-test-slave-vrq1"
server_key: "44f60dee9efd6481d582745e7e5fc467"
server_key_iv: "c90f5450e9b7d97c96b95922aa991e3e"
server_key_version: "encryptionkey@0"
I20260621 05:10:34.749715 6865 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782018616648578-3598-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782018616648578-3598-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782018616648578-3598-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 05:10:34.770366 6865 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 05:10:34.773078 6865 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 05:10:34.773305 6865 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 05:10:34.773566 6865 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 05:10:34.773905 6865 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 05:10:34.773962 6865 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 05:10:34.773998 6865 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 05:10:34.774045 6865 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 05:10:34.780385 6865 rpc_server.cc:307] RPC server started. Bound to: 127.3.131.130:46529
I20260621 05:10:34.780433 6993 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.131.130:46529 every 8 connection(s)
I20260621 05:10:34.780792 6865 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782018616648578-3598-0/minicluster-data/ts-1/data/info.pb
Jun 21 05:10:34 dist-test-slave-vrq1 krb5kdc[6624](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782018634, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.131.130@KRBTEST.COM for kudu/127.3.131.190@KRBTEST.COM
I20260621 05:10:34.789415 3598 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWZOx2_/build/release/bin/kudu as pid 6865
I20260621 05:10:34.789577 3598 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782018616648578-3598-0/minicluster-data/ts-1/wal/instance
I20260621 05:10:34.789810 3598 external_mini_cluster.cc:1428] Setting key 6edc27c4b4d74eabffa85e745475ee4d
I20260621 05:10:34.791453 6709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:10:34.781695 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.131.130:59353 (local address 127.3.131.190:42441)
0621 05:10:34.781763 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0621 05:10:34.781765 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:10:34.781986 (+ 221us) server_negotiation.cc:407] Connection header received
0621 05:10:34.782531 (+ 545us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:10:34.782533 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:10:34.782546 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:10:34.782594 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:10:34.783754 (+ 1160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:10:34.784310 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:10:34.785374 (+ 1064us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:10:34.785580 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:10:34.785648 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:10:34.788087 (+ 2439us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:10:34.788089 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:10:34.788089 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:10:34.788097 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:10:34.789644 (+ 1547us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:10:34.790173 (+ 529us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:10:34.790174 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:10:34.790174 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:10:34.790235 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:10:34.790633 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:10:34.790633 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:10:34.790634 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:10:34.790829 (+ 195us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:10:34.790901 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0621 05:10:34.791213 (+ 312us) server_negotiation.cc:299] Negotiation successful
0621 05:10:34.791304 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260621 05:10:34.791530 6996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:10:34.781711 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.131.190:42441 (local address 127.3.131.130:59353)
0621 05:10:34.781861 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0621 05:10:34.781899 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0621 05:10:34.782435 (+ 536us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:10:34.782659 (+ 224us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:10:34.782663 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:10:34.783408 (+ 745us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:10:34.783630 (+ 222us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:10:34.783636 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:10:34.784428 (+ 792us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:10:34.784429 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:10:34.785261 (+ 832us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:10:34.785263 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:10:34.785682 (+ 419us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:10:34.785683 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:10:34.785800 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:10:34.786403 (+ 603us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 05:10:34.786412 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:10:34.787947 (+ 1535us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 05:10:34.789738 (+ 1791us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:10:34.789744 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:10:34.789746 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:10:34.790084 (+ 338us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:10:34.790396 (+ 312us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:10:34.790397 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:10:34.790398 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:10:34.790551 (+ 153us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:10:34.790902 (+ 351us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 05:10:34.790908 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 05:10:34.791039 (+ 131us) client_negotiation.cc:769] Sending connection context
0621 05:10:34.791128 (+ 89us) client_negotiation.cc:240] Negotiation successful
0621 05:10:34.791335 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":80,"threads_started":1}
I20260621 05:10:34.792089 6994 heartbeater.cc:347] Connected to a master server at 127.3.131.190:42441
I20260621 05:10:34.792237 6994 heartbeater.cc:464] Registering TS with master...
I20260621 05:10:34.792479 6994 heartbeater.cc:511] Master 127.3.131.190:42441 requested a full tablet report, sending...
I20260621 05:10:34.792914 6669 ts_manager.cc:194] Registered new tserver with Master: 18455b1378024adf9a4f5092a093eb54 (127.3.131.130:46529)
I20260621 05:10:34.793327 6669 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.131.130@KRBTEST.COM'} at 127.3.131.130:59353
WARNING: no policy specified for kudu/127.3.131.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.131.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.131.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782018616648578-3598-0/krb5kdc/kudu_127.3.131.131.keytab.
Entry for principal kudu/127.3.131.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782018616648578-3598-0/krb5kdc/kudu_127.3.131.131.keytab.
I20260621 05:10:34.823246 3598 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWZOx2_/build/release/bin/kudu
/tmp/dist-test-taskWZOx2_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782018616648578-3598-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782018616648578-3598-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782018616648578-3598-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782018616648578-3598-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.131.131:0
--local_ip_for_outbound_sockets=127.3.131.131
--webserver_interface=127.3.131.131
--webserver_port=0
--tserver_master_addrs=127.3.131.190:42441
--keytab_file=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782018616648578-3598-0/krb5kdc/kudu_127.3.131.131.keytab
--principal=kudu/127.3.131.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.131.148:45413
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782018616648578-3598-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782018616648578-3598-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782018616648578-3598-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 05:10:34.916164 6999 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 05:10:34.916428 6999 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 05:10:34.916476 6999 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 05:10:34.918002 6999 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 05:10:34.918092 6999 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 05:10:34.918149 6999 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.131.131
I20260621 05:10:34.920065 6999 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.131.148:45413
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782018616648578-3598-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782018616648578-3598-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782018616648578-3598-0/krb5kdc/kudu_127.3.131.131.keytab
--principal=kudu/127.3.131.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.131.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782018616648578-3598-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.131.131
--webserver_port=0
--tserver_master_addrs=127.3.131.190:42441
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6999
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.131.131
--log_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782018616648578-3598-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 21 Jun 2026 04:16:16 UTC on ccycloud.quasar-xycaha.root.comops.site
build id 4457881
I20260621 05:10:34.920378 6999 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 05:10:34.920656 6999 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 05:10:34.926822 7007 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:10:34.926868 7004 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:10:34.926852 7005 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:10:34.927269 6999 server_base.cc:1034] running on GCE node
I20260621 05:10:34.927520 6999 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 05:10:34.927794 6999 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 05:10:34.928977 6999 hybrid_clock.cc:648] HybridClock initialized: now 1782018634928957 us; error 26 us; skew 500 ppm
Jun 21 05:10:34 dist-test-slave-vrq1 krb5kdc[6624](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782018634, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.131.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 05:10:34.931879 6999 init.cc:377] Logged in from keytab as kudu/127.3.131.131@KRBTEST.COM (short username kudu)
I20260621 05:10:34.932452 6999 webserver.cc:460] Webserver started at http://127.3.131.131:39131/ using document root <none> and password file <none>
I20260621 05:10:34.932713 6999 fs_manager.cc:359] Metadata directory not provided
I20260621 05:10:34.932783 6999 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 05:10:34.932931 6999 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 05:10:34.934088 6999 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782018616648578-3598-0/minicluster-data/ts-2/data/instance:
uuid: "5a18b2a865a648beaf9955c26cd3a0ad"
format_stamp: "Formatted at 2026-06-21 05:10:34 on dist-test-slave-vrq1"
server_key: "01b1f46bf9eac4658f09b0ad2a5494dc"
server_key_iv: "ba6f8d5d4dbb44f32a3bb0a96f1c1f27"
server_key_version: "encryptionkey@0"
I20260621 05:10:34.934466 6999 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782018616648578-3598-0/minicluster-data/ts-2/wal/instance:
uuid: "5a18b2a865a648beaf9955c26cd3a0ad"
format_stamp: "Formatted at 2026-06-21 05:10:34 on dist-test-slave-vrq1"
server_key: "01b1f46bf9eac4658f09b0ad2a5494dc"
server_key_iv: "ba6f8d5d4dbb44f32a3bb0a96f1c1f27"
server_key_version: "encryptionkey@0"
I20260621 05:10:34.936054 6999 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260621 05:10:34.937006 7014 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 05:10:34.937336 6999 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260621 05:10:34.937505 6999 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782018616648578-3598-0/minicluster-data/ts-2/data,/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782018616648578-3598-0/minicluster-data/ts-2/wal
uuid: "5a18b2a865a648beaf9955c26cd3a0ad"
format_stamp: "Formatted at 2026-06-21 05:10:34 on dist-test-slave-vrq1"
server_key: "01b1f46bf9eac4658f09b0ad2a5494dc"
server_key_iv: "ba6f8d5d4dbb44f32a3bb0a96f1c1f27"
server_key_version: "encryptionkey@0"
I20260621 05:10:34.937616 6999 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782018616648578-3598-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782018616648578-3598-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782018616648578-3598-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 05:10:34.946189 6999 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 05:10:34.948426 6999 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 05:10:34.948601 6999 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 05:10:34.948853 6999 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 05:10:34.949182 6999 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 05:10:34.949234 6999 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 05:10:34.949276 6999 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 05:10:34.949306 6999 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 05:10:34.955971 6999 rpc_server.cc:307] RPC server started. Bound to: 127.3.131.131:36843
I20260621 05:10:34.956043 7127 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.131.131:36843 every 8 connection(s)
I20260621 05:10:34.956566 6999 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782018616648578-3598-0/minicluster-data/ts-2/data/info.pb
I20260621 05:10:34.958587 3598 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWZOx2_/build/release/bin/kudu as pid 6999
I20260621 05:10:34.958658 3598 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782018616648578-3598-0/minicluster-data/ts-2/wal/instance
I20260621 05:10:34.958846 3598 external_mini_cluster.cc:1428] Setting key 2b9bde41d3c0ee4fa5239a87007ebef6
Jun 21 05:10:34 dist-test-slave-vrq1 krb5kdc[6624](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782018634, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.131.131@KRBTEST.COM for kudu/127.3.131.190@KRBTEST.COM
I20260621 05:10:34.966516 6709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:10:34.957526 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.131.131:45641 (local address 127.3.131.190:42441)
0621 05:10:34.957626 (+ 100us) server_negotiation.cc:206] Beginning negotiation
0621 05:10:34.957629 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0621 05:10:34.957780 (+ 151us) server_negotiation.cc:407] Connection header received
0621 05:10:34.958260 (+ 480us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:10:34.958262 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:10:34.958273 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:10:34.958344 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:10:34.959476 (+ 1132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:10:34.959995 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:10:34.961032 (+ 1037us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:10:34.961249 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:10:34.961335 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:10:34.963735 (+ 2400us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:10:34.963737 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:10:34.963737 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:10:34.963744 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:10:34.965065 (+ 1321us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:10:34.965506 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:10:34.965507 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:10:34.965508 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:10:34.965571 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:10:34.965850 (+ 279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:10:34.965851 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:10:34.965851 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:10:34.965972 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:10:34.966050 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0621 05:10:34.966334 (+ 284us) server_negotiation.cc:299] Negotiation successful
0621 05:10:34.966419 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260621 05:10:34.966732 7130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:10:34.957460 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.131.190:42441 (local address 127.3.131.131:45641)
0621 05:10:34.957625 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0621 05:10:34.957668 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0621 05:10:34.958146 (+ 478us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:10:34.958402 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:10:34.958406 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:10:34.959096 (+ 690us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:10:34.959372 (+ 276us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:10:34.959379 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:10:34.960095 (+ 716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:10:34.960096 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:10:34.960927 (+ 831us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:10:34.960929 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:10:34.961345 (+ 416us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:10:34.961346 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:10:34.961489 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:10:34.962088 (+ 599us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 05:10:34.962097 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:10:34.963588 (+ 1491us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 05:10:34.965175 (+ 1587us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:10:34.965183 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:10:34.965185 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:10:34.965418 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:10:34.965655 (+ 237us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:10:34.965655 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:10:34.965656 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:10:34.965773 (+ 117us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:10:34.966039 (+ 266us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 05:10:34.966045 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 05:10:34.966201 (+ 156us) client_negotiation.cc:769] Sending connection context
0621 05:10:34.966315 (+ 114us) client_negotiation.cc:240] Negotiation successful
0621 05:10:34.966483 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":93,"threads_started":1}
I20260621 05:10:34.967231 7128 heartbeater.cc:347] Connected to a master server at 127.3.131.190:42441
I20260621 05:10:34.967357 7128 heartbeater.cc:464] Registering TS with master...
I20260621 05:10:34.967543 7128 heartbeater.cc:511] Master 127.3.131.190:42441 requested a full tablet report, sending...
I20260621 05:10:34.967955 6669 ts_manager.cc:194] Registered new tserver with Master: 5a18b2a865a648beaf9955c26cd3a0ad (127.3.131.131:36843)
I20260621 05:10:34.968336 6669 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.131.131@KRBTEST.COM'} at 127.3.131.131:45641
I20260621 05:10:34.970593 3598 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 21 05:10:34 dist-test-slave-vrq1 krb5kdc[6624](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782018634, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Jun 21 05:10:34 dist-test-slave-vrq1 krb5kdc[6624](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782018634, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.3.131.190@KRBTEST.COM
I20260621 05:10:34.995832 6709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:10:34.987483 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48142 (local address 127.3.131.190:42441)
0621 05:10:34.987587 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0621 05:10:34.987589 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:10:34.987599 (+ 10us) server_negotiation.cc:407] Connection header received
0621 05:10:34.987669 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:10:34.987670 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:10:34.987678 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:10:34.987745 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:10:34.988485 (+ 740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:10:34.989045 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:10:34.989851 (+ 806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:10:34.990081 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:10:34.990152 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:10:34.992803 (+ 2651us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:10:34.992805 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:10:34.992806 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:10:34.992814 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:10:34.994501 (+ 1687us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:10:34.994874 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:10:34.994875 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:10:34.994875 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:10:34.994930 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:10:34.995285 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:10:34.995286 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:10:34.995287 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:10:34.995454 (+ 167us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:10:34.995559 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0621 05:10:34.995681 (+ 122us) server_negotiation.cc:299] Negotiation successful
0621 05:10:34.995763 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
W20260621 05:10:34.996115 6669 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:48142
I20260621 05:10:34.996881 3598 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWZOx2_/build/release/bin/kudu with pid 6731
I20260621 05:10:35.001726 3598 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWZOx2_/build/release/bin/kudu with pid 6865
I20260621 05:10:35.006567 3598 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWZOx2_/build/release/bin/kudu with pid 6999
I20260621 05:10:35.011029 3598 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWZOx2_/build/release/bin/kudu with pid 6637
2026-06-21T05:10:35Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (2027 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782018616648578-3598-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 21 05:10:35 dist-test-slave-vrq1 krb5kdc[7140](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 21 05:10:35 dist-test-slave-vrq1 krb5kdc[7140](info): set up 2 sockets
Jun 21 05:10:35 dist-test-slave-vrq1 krb5kdc[7140](info): commencing operation
krb5kdc: starting...
W20260621 05:10:36.067695 3598 mini_kdc.cc:121] Time spent starting KDC: real 1.036s user 0.001s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782018616648578-3598-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782018616648578-3598-0/krb5kdc/test-user.keytab.
Jun 21 05:10:36 dist-test-slave-vrq1 krb5kdc[7140](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782018636, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-21T05:10:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-21T05:10:36Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.131.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.131.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.131.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782018616648578-3598-0/krb5kdc/kudu_127.3.131.190.keytab.
Entry for principal kudu/127.3.131.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782018616648578-3598-0/krb5kdc/kudu_127.3.131.190.keytab.
I20260621 05:10:36.199115 3598 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWZOx2_/build/release/bin/kudu
/tmp/dist-test-taskWZOx2_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782018616648578-3598-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782018616648578-3598-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782018616648578-3598-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782018616648578-3598-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.131.190:36241
--webserver_interface=127.3.131.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782018616648578-3598-0/krb5kdc/kudu_127.3.131.190.keytab
--principal=kudu/127.3.131.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.131.148:41969
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.131.190:36241
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782018616648578-3598-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782018616648578-3598-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782018616648578-3598-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 05:10:36.282841 7153 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 05:10:36.283133 7153 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 05:10:36.283201 7153 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 05:10:36.284463 7153 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 05:10:36.284540 7153 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 05:10:36.284569 7153 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 05:10:36.284587 7153 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 05:10:36.284605 7153 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 05:10:36.286201 7153 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.131.148:41969
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782018616648578-3598-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782018616648578-3598-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.131.190:36241
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782018616648578-3598-0/krb5kdc/kudu_127.3.131.190.keytab
--principal=kudu/127.3.131.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.131.190:36241
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782018616648578-3598-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.131.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7153
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782018616648578-3598-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 21 Jun 2026 04:16:16 UTC on ccycloud.quasar-xycaha.root.comops.site
build id 4457881
I20260621 05:10:36.286495 7153 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 05:10:36.286733 7153 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 05:10:36.291985 7161 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:10:36.291985 7158 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:10:36.292135 7153 server_base.cc:1034] running on GCE node
W20260621 05:10:36.293620 7159 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:10:36.294034 7153 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 05:10:36.294391 7153 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 05:10:36.295620 7153 hybrid_clock.cc:648] HybridClock initialized: now 1782018636295591 us; error 42 us; skew 500 ppm
Jun 21 05:10:36 dist-test-slave-vrq1 krb5kdc[7140](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782018636, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.131.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 05:10:36.298482 7153 init.cc:377] Logged in from keytab as kudu/127.3.131.190@KRBTEST.COM (short username kudu)
I20260621 05:10:36.298929 7153 webserver.cc:460] Webserver started at http://127.3.131.190:42367/ using document root <none> and password file <none>
I20260621 05:10:36.299135 7153 fs_manager.cc:359] Metadata directory not provided
I20260621 05:10:36.299247 7153 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 05:10:36.299362 7153 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 05:10:36.300457 7153 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782018616648578-3598-0/minicluster-data/master-0/data/instance:
uuid: "249ed2a1185f440a9fe434b3ec47e9a8"
format_stamp: "Formatted at 2026-06-21 05:10:36 on dist-test-slave-vrq1"
server_key: "9dce4920eae8682373449e9b1f655c12"
server_key_iv: "566022738a7f922453a507cc39a9ad54"
server_key_version: "encryptionkey@0"
I20260621 05:10:36.300812 7153 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782018616648578-3598-0/minicluster-data/master-0/wal/instance:
uuid: "249ed2a1185f440a9fe434b3ec47e9a8"
format_stamp: "Formatted at 2026-06-21 05:10:36 on dist-test-slave-vrq1"
server_key: "9dce4920eae8682373449e9b1f655c12"
server_key_iv: "566022738a7f922453a507cc39a9ad54"
server_key_version: "encryptionkey@0"
I20260621 05:10:36.302270 7153 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260621 05:10:36.303388 7168 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 05:10:36.304039 7153 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260621 05:10:36.304154 7153 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782018616648578-3598-0/minicluster-data/master-0/data,/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782018616648578-3598-0/minicluster-data/master-0/wal
uuid: "249ed2a1185f440a9fe434b3ec47e9a8"
format_stamp: "Formatted at 2026-06-21 05:10:36 on dist-test-slave-vrq1"
server_key: "9dce4920eae8682373449e9b1f655c12"
server_key_iv: "566022738a7f922453a507cc39a9ad54"
server_key_version: "encryptionkey@0"
I20260621 05:10:36.304437 7153 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782018616648578-3598-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782018616648578-3598-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782018616648578-3598-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 05:10:36.314186 7153 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 05:10:36.316989 7153 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 05:10:36.317131 7153 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 05:10:36.321828 7153 rpc_server.cc:307] RPC server started. Bound to: 127.3.131.190:36241
I20260621 05:10:36.322139 7220 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.131.190:36241 every 8 connection(s)
I20260621 05:10:36.322315 7153 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782018616648578-3598-0/minicluster-data/master-0/data/info.pb
I20260621 05:10:36.323110 7221 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 05:10:36.325132 3598 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWZOx2_/build/release/bin/kudu as pid 7153
I20260621 05:10:36.325273 3598 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782018616648578-3598-0/minicluster-data/master-0/wal/instance
I20260621 05:10:36.325519 3598 external_mini_cluster.cc:1428] Setting key b7e4630ac0c24209596eb4b1354f7638
I20260621 05:10:36.325878 7221 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 249ed2a1185f440a9fe434b3ec47e9a8: Bootstrap starting.
I20260621 05:10:36.326644 7221 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 249ed2a1185f440a9fe434b3ec47e9a8: Neither blocks nor log segments found. Creating new log.
I20260621 05:10:36.326970 7221 log.cc:824] T 00000000000000000000000000000000 P 249ed2a1185f440a9fe434b3ec47e9a8: Log is configured to *not* fsync() on all Append() calls
I20260621 05:10:36.327874 7221 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 249ed2a1185f440a9fe434b3ec47e9a8: No bootstrap required, opened a new log
I20260621 05:10:36.330135 7221 raft_consensus.cc:348] T 00000000000000000000000000000000 P 249ed2a1185f440a9fe434b3ec47e9a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "249ed2a1185f440a9fe434b3ec47e9a8" member_type: VOTER last_known_addr { host: "127.3.131.190" port: 36241 } }
I20260621 05:10:36.330518 7221 raft_consensus.cc:374] T 00000000000000000000000000000000 P 249ed2a1185f440a9fe434b3ec47e9a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 05:10:36.330582 7221 raft_consensus.cc:729] T 00000000000000000000000000000000 P 249ed2a1185f440a9fe434b3ec47e9a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 249ed2a1185f440a9fe434b3ec47e9a8, State: Initialized, Role: FOLLOWER
I20260621 05:10:36.330765 7221 consensus_queue.cc:260] T 00000000000000000000000000000000 P 249ed2a1185f440a9fe434b3ec47e9a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "249ed2a1185f440a9fe434b3ec47e9a8" member_type: VOTER last_known_addr { host: "127.3.131.190" port: 36241 } }
I20260621 05:10:36.330883 7221 raft_consensus.cc:388] T 00000000000000000000000000000000 P 249ed2a1185f440a9fe434b3ec47e9a8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260621 05:10:36.330929 7221 raft_consensus.cc:482] T 00000000000000000000000000000000 P 249ed2a1185f440a9fe434b3ec47e9a8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260621 05:10:36.330979 7221 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 249ed2a1185f440a9fe434b3ec47e9a8 [term 0 FOLLOWER]: Advancing to term 1
I20260621 05:10:36.331992 7221 raft_consensus.cc:504] T 00000000000000000000000000000000 P 249ed2a1185f440a9fe434b3ec47e9a8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "249ed2a1185f440a9fe434b3ec47e9a8" member_type: VOTER last_known_addr { host: "127.3.131.190" port: 36241 } }
I20260621 05:10:36.332139 7221 leader_election.cc:302] T 00000000000000000000000000000000 P 249ed2a1185f440a9fe434b3ec47e9a8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 249ed2a1185f440a9fe434b3ec47e9a8; no voters:
I20260621 05:10:36.332327 7221 leader_election.cc:288] T 00000000000000000000000000000000 P 249ed2a1185f440a9fe434b3ec47e9a8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260621 05:10:36.332388 7226 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 249ed2a1185f440a9fe434b3ec47e9a8 [term 1 FOLLOWER]: Leader election won for term 1
I20260621 05:10:36.332578 7226 raft_consensus.cc:686] T 00000000000000000000000000000000 P 249ed2a1185f440a9fe434b3ec47e9a8 [term 1 LEADER]: Becoming Leader. State: Replica: 249ed2a1185f440a9fe434b3ec47e9a8, State: Running, Role: LEADER
I20260621 05:10:36.332762 7226 consensus_queue.cc:237] T 00000000000000000000000000000000 P 249ed2a1185f440a9fe434b3ec47e9a8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "249ed2a1185f440a9fe434b3ec47e9a8" member_type: VOTER last_known_addr { host: "127.3.131.190" port: 36241 } }
I20260621 05:10:36.332639 7221 sys_catalog.cc:564] T 00000000000000000000000000000000 P 249ed2a1185f440a9fe434b3ec47e9a8 [sys.catalog]: configured and running, proceeding with master startup.
I20260621 05:10:36.333242 7228 sys_catalog.cc:455] T 00000000000000000000000000000000 P 249ed2a1185f440a9fe434b3ec47e9a8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "249ed2a1185f440a9fe434b3ec47e9a8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "249ed2a1185f440a9fe434b3ec47e9a8" member_type: VOTER last_known_addr { host: "127.3.131.190" port: 36241 } } }
I20260621 05:10:36.333253 7227 sys_catalog.cc:455] T 00000000000000000000000000000000 P 249ed2a1185f440a9fe434b3ec47e9a8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 249ed2a1185f440a9fe434b3ec47e9a8. Latest consensus state: current_term: 1 leader_uuid: "249ed2a1185f440a9fe434b3ec47e9a8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "249ed2a1185f440a9fe434b3ec47e9a8" member_type: VOTER last_known_addr { host: "127.3.131.190" port: 36241 } } }
I20260621 05:10:36.333343 7228 sys_catalog.cc:458] T 00000000000000000000000000000000 P 249ed2a1185f440a9fe434b3ec47e9a8 [sys.catalog]: This master's current role is: LEADER
I20260621 05:10:36.333375 7227 sys_catalog.cc:458] T 00000000000000000000000000000000 P 249ed2a1185f440a9fe434b3ec47e9a8 [sys.catalog]: This master's current role is: LEADER
Jun 21 05:10:36 dist-test-slave-vrq1 krb5kdc[7140](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782018636, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.131.190@KRBTEST.COM
I20260621 05:10:36.334667 7230 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260621 05:10:36.335249 7230 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260621 05:10:36.337252 7230 catalog_manager.cc:1347] Generated new cluster ID: 71bf3324ae7148db8dfc35cb1328c263
I20260621 05:10:36.337318 7230 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260621 05:10:36.338645 7224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:10:36.326151 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55634 (local address 127.3.131.190:36241)
0621 05:10:36.326376 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0621 05:10:36.326385 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0621 05:10:36.326427 (+ 42us) server_negotiation.cc:407] Connection header received
0621 05:10:36.326976 (+ 549us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:10:36.326984 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:10:36.327074 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:10:36.327301 (+ 227us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:10:36.328107 (+ 806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:10:36.329135 (+ 1028us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:10:36.330476 (+ 1341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:10:36.331251 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:10:36.331363 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:10:36.334520 (+ 3157us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:10:36.334525 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:10:36.334525 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:10:36.334540 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:10:36.336774 (+ 2234us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:10:36.337243 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:10:36.337246 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:10:36.337246 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:10:36.337416 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:10:36.337675 (+ 259us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:10:36.337676 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:10:36.337677 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:10:36.337939 (+ 262us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:10:36.338124 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0621 05:10:36.338261 (+ 137us) server_negotiation.cc:299] Negotiation successful
0621 05:10:36.338406 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":166,"thread_start_us":130,"threads_started":1}
I20260621 05:10:36.345330 7230 catalog_manager.cc:1370] Generated new certificate authority record
I20260621 05:10:36.345885 7230 catalog_manager.cc:1504] Loading token signing keys...
I20260621 05:10:36.356565 7230 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 249ed2a1185f440a9fe434b3ec47e9a8: Generated new TSK 0
I20260621 05:10:36.356829 7230 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.3.131.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.131.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.131.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782018616648578-3598-0/krb5kdc/kudu_127.3.131.129.keytab.
Entry for principal kudu/127.3.131.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782018616648578-3598-0/krb5kdc/kudu_127.3.131.129.keytab.
I20260621 05:10:36.395051 3598 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWZOx2_/build/release/bin/kudu
/tmp/dist-test-taskWZOx2_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782018616648578-3598-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782018616648578-3598-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782018616648578-3598-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782018616648578-3598-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.131.129:0
--local_ip_for_outbound_sockets=127.3.131.129
--webserver_interface=127.3.131.129
--webserver_port=0
--tserver_master_addrs=127.3.131.190:36241
--keytab_file=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782018616648578-3598-0/krb5kdc/kudu_127.3.131.129.keytab
--principal=kudu/127.3.131.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.131.148:41969
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782018616648578-3598-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782018616648578-3598-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782018616648578-3598-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 05:10:36.495492 7247 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 05:10:36.495766 7247 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 05:10:36.495808 7247 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 05:10:36.497161 7247 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 05:10:36.497225 7247 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 05:10:36.497275 7247 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.131.129
I20260621 05:10:36.498906 7247 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.131.148:41969
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782018616648578-3598-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782018616648578-3598-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782018616648578-3598-0/krb5kdc/kudu_127.3.131.129.keytab
--principal=kudu/127.3.131.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.131.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782018616648578-3598-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.131.129
--webserver_port=0
--tserver_master_addrs=127.3.131.190:36241
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7247
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.131.129
--log_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782018616648578-3598-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 21 Jun 2026 04:16:16 UTC on ccycloud.quasar-xycaha.root.comops.site
build id 4457881
I20260621 05:10:36.499251 7247 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 05:10:36.499495 7247 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 05:10:36.505148 7253 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:10:36.505244 7255 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:10:36.505255 7252 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:10:36.505431 7247 server_base.cc:1034] running on GCE node
I20260621 05:10:36.505686 7247 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 05:10:36.505906 7247 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 05:10:36.507094 7247 hybrid_clock.cc:648] HybridClock initialized: now 1782018636507076 us; error 29 us; skew 500 ppm
Jun 21 05:10:36 dist-test-slave-vrq1 krb5kdc[7140](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782018636, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.131.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 05:10:36.509907 7247 init.cc:377] Logged in from keytab as kudu/127.3.131.129@KRBTEST.COM (short username kudu)
I20260621 05:10:36.510408 7247 webserver.cc:460] Webserver started at http://127.3.131.129:39245/ using document root <none> and password file <none>
I20260621 05:10:36.510651 7247 fs_manager.cc:359] Metadata directory not provided
I20260621 05:10:36.510723 7247 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 05:10:36.510882 7247 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 05:10:36.512054 7247 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782018616648578-3598-0/minicluster-data/ts-0/data/instance:
uuid: "daaa20948e0e458d8787a6f30e1daa11"
format_stamp: "Formatted at 2026-06-21 05:10:36 on dist-test-slave-vrq1"
server_key: "b403f4f97de2b9f256904e8b9b0dbe83"
server_key_iv: "a28cd09dfb1ba2bd7e8e4832e20569e6"
server_key_version: "encryptionkey@0"
I20260621 05:10:36.512449 7247 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782018616648578-3598-0/minicluster-data/ts-0/wal/instance:
uuid: "daaa20948e0e458d8787a6f30e1daa11"
format_stamp: "Formatted at 2026-06-21 05:10:36 on dist-test-slave-vrq1"
server_key: "b403f4f97de2b9f256904e8b9b0dbe83"
server_key_iv: "a28cd09dfb1ba2bd7e8e4832e20569e6"
server_key_version: "encryptionkey@0"
I20260621 05:10:36.513923 7247 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260621 05:10:36.514899 7262 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 05:10:36.515100 7247 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260621 05:10:36.515275 7247 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782018616648578-3598-0/minicluster-data/ts-0/data,/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782018616648578-3598-0/minicluster-data/ts-0/wal
uuid: "daaa20948e0e458d8787a6f30e1daa11"
format_stamp: "Formatted at 2026-06-21 05:10:36 on dist-test-slave-vrq1"
server_key: "b403f4f97de2b9f256904e8b9b0dbe83"
server_key_iv: "a28cd09dfb1ba2bd7e8e4832e20569e6"
server_key_version: "encryptionkey@0"
I20260621 05:10:36.515367 7247 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782018616648578-3598-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782018616648578-3598-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782018616648578-3598-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 05:10:36.532639 7247 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 05:10:36.535251 7247 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 05:10:36.535434 7247 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 05:10:36.535702 7247 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 05:10:36.536080 7247 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 05:10:36.536139 7247 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 05:10:36.536182 7247 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 05:10:36.536238 7247 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 05:10:36.542387 7247 rpc_server.cc:307] RPC server started. Bound to: 127.3.131.129:35745
I20260621 05:10:36.542450 7375 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.131.129:35745 every 8 connection(s)
I20260621 05:10:36.542809 7247 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782018616648578-3598-0/minicluster-data/ts-0/data/info.pb
Jun 21 05:10:36 dist-test-slave-vrq1 krb5kdc[7140](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782018636, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.131.129@KRBTEST.COM for kudu/127.3.131.190@KRBTEST.COM
I20260621 05:10:36.551748 3598 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWZOx2_/build/release/bin/kudu as pid 7247
I20260621 05:10:36.551898 3598 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782018616648578-3598-0/minicluster-data/ts-0/wal/instance
I20260621 05:10:36.552166 3598 external_mini_cluster.cc:1428] Setting key 9e29ded357c893d87cba64a1b12794a9
I20260621 05:10:36.554347 7224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:10:36.543690 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.131.129:33619 (local address 127.3.131.190:36241)
0621 05:10:36.543812 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0621 05:10:36.543814 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:10:36.544007 (+ 193us) server_negotiation.cc:407] Connection header received
0621 05:10:36.544971 (+ 964us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:10:36.544972 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:10:36.544984 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:10:36.545049 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:10:36.545837 (+ 788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:10:36.546378 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:10:36.547627 (+ 1249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:10:36.547835 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:10:36.547901 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:10:36.550530 (+ 2629us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:10:36.550533 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:10:36.550533 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:10:36.550543 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:10:36.552461 (+ 1918us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:10:36.552995 (+ 534us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:10:36.552996 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:10:36.552997 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:10:36.553220 (+ 223us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:10:36.553584 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:10:36.553585 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:10:36.553586 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:10:36.553732 (+ 146us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:10:36.553822 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0621 05:10:36.554138 (+ 316us) server_negotiation.cc:299] Negotiation successful
0621 05:10:36.554249 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
I20260621 05:10:36.554589 7378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:10:36.543618 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.131.190:36241 (local address 127.3.131.129:33619)
0621 05:10:36.543886 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0621 05:10:36.543933 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0621 05:10:36.544873 (+ 940us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:10:36.545179 (+ 306us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:10:36.545183 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:10:36.545499 (+ 316us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:10:36.545705 (+ 206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:10:36.545710 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:10:36.546509 (+ 799us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:10:36.546510 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:10:36.547522 (+ 1012us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:10:36.547525 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:10:36.547950 (+ 425us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:10:36.547951 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:10:36.548108 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:10:36.548693 (+ 585us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 05:10:36.548705 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:10:36.550350 (+ 1645us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 05:10:36.552561 (+ 2211us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:10:36.552572 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:10:36.552580 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:10:36.552877 (+ 297us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:10:36.553335 (+ 458us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:10:36.553336 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:10:36.553337 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:10:36.553494 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:10:36.553814 (+ 320us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 05:10:36.553822 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 05:10:36.553993 (+ 171us) client_negotiation.cc:769] Sending connection context
0621 05:10:36.554116 (+ 123us) client_negotiation.cc:240] Negotiation successful
0621 05:10:36.554313 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":229,"thread_start_us":77,"threads_started":1}
I20260621 05:10:36.555285 7376 heartbeater.cc:347] Connected to a master server at 127.3.131.190:36241
I20260621 05:10:36.555388 7376 heartbeater.cc:464] Registering TS with master...
I20260621 05:10:36.555619 7376 heartbeater.cc:511] Master 127.3.131.190:36241 requested a full tablet report, sending...
I20260621 05:10:36.556409 7185 ts_manager.cc:194] Registered new tserver with Master: daaa20948e0e458d8787a6f30e1daa11 (127.3.131.129:35745)
I20260621 05:10:36.557124 7185 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.131.129@KRBTEST.COM'} at 127.3.131.129:33619
WARNING: no policy specified for kudu/127.3.131.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.131.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.131.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782018616648578-3598-0/krb5kdc/kudu_127.3.131.130.keytab.
Entry for principal kudu/127.3.131.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782018616648578-3598-0/krb5kdc/kudu_127.3.131.130.keytab.
I20260621 05:10:36.586831 3598 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWZOx2_/build/release/bin/kudu
/tmp/dist-test-taskWZOx2_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782018616648578-3598-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782018616648578-3598-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782018616648578-3598-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782018616648578-3598-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.131.130:0
--local_ip_for_outbound_sockets=127.3.131.130
--webserver_interface=127.3.131.130
--webserver_port=0
--tserver_master_addrs=127.3.131.190:36241
--keytab_file=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782018616648578-3598-0/krb5kdc/kudu_127.3.131.130.keytab
--principal=kudu/127.3.131.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.131.148:41969
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782018616648578-3598-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782018616648578-3598-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782018616648578-3598-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 05:10:36.677742 7381 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 05:10:36.678012 7381 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 05:10:36.678042 7381 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 05:10:36.679621 7381 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 05:10:36.679792 7381 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 05:10:36.679867 7381 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.131.130
I20260621 05:10:36.682068 7381 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.131.148:41969
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782018616648578-3598-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782018616648578-3598-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782018616648578-3598-0/krb5kdc/kudu_127.3.131.130.keytab
--principal=kudu/127.3.131.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.131.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782018616648578-3598-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.131.130
--webserver_port=0
--tserver_master_addrs=127.3.131.190:36241
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7381
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.131.130
--log_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782018616648578-3598-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 21 Jun 2026 04:16:16 UTC on ccycloud.quasar-xycaha.root.comops.site
build id 4457881
I20260621 05:10:36.682353 7381 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 05:10:36.682585 7381 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 05:10:36.689002 7387 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:10:36.689002 7389 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:10:36.689018 7386 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:10:36.689944 7381 server_base.cc:1034] running on GCE node
I20260621 05:10:36.690174 7381 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 05:10:36.690402 7381 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 05:10:36.691566 7381 hybrid_clock.cc:648] HybridClock initialized: now 1782018636691539 us; error 37 us; skew 500 ppm
Jun 21 05:10:36 dist-test-slave-vrq1 krb5kdc[7140](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782018636, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.131.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 05:10:36.695351 7381 init.cc:377] Logged in from keytab as kudu/127.3.131.130@KRBTEST.COM (short username kudu)
I20260621 05:10:36.695897 7381 webserver.cc:460] Webserver started at http://127.3.131.130:44345/ using document root <none> and password file <none>
I20260621 05:10:36.696178 7381 fs_manager.cc:359] Metadata directory not provided
I20260621 05:10:36.696427 7381 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 05:10:36.696638 7381 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 05:10:36.698134 7381 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782018616648578-3598-0/minicluster-data/ts-1/data/instance:
uuid: "76c09b10e2df47149979349cd84b2882"
format_stamp: "Formatted at 2026-06-21 05:10:36 on dist-test-slave-vrq1"
server_key: "f03568df5730668d12c87b196140034b"
server_key_iv: "f2f8f23ca571b5b42f826fe41590ba98"
server_key_version: "encryptionkey@0"
I20260621 05:10:36.698591 7381 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782018616648578-3598-0/minicluster-data/ts-1/wal/instance:
uuid: "76c09b10e2df47149979349cd84b2882"
format_stamp: "Formatted at 2026-06-21 05:10:36 on dist-test-slave-vrq1"
server_key: "f03568df5730668d12c87b196140034b"
server_key_iv: "f2f8f23ca571b5b42f826fe41590ba98"
server_key_version: "encryptionkey@0"
I20260621 05:10:36.700601 7381 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.004s sys 0.000s
I20260621 05:10:36.701681 7396 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 05:10:36.701946 7381 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260621 05:10:36.702173 7381 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782018616648578-3598-0/minicluster-data/ts-1/data,/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782018616648578-3598-0/minicluster-data/ts-1/wal
uuid: "76c09b10e2df47149979349cd84b2882"
format_stamp: "Formatted at 2026-06-21 05:10:36 on dist-test-slave-vrq1"
server_key: "f03568df5730668d12c87b196140034b"
server_key_iv: "f2f8f23ca571b5b42f826fe41590ba98"
server_key_version: "encryptionkey@0"
I20260621 05:10:36.702267 7381 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782018616648578-3598-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782018616648578-3598-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782018616648578-3598-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 05:10:36.716374 7381 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 05:10:36.719053 7381 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 05:10:36.719313 7381 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 05:10:36.719619 7381 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 05:10:36.719944 7381 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 05:10:36.719998 7381 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 05:10:36.720043 7381 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 05:10:36.720072 7381 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 05:10:36.726246 7381 rpc_server.cc:307] RPC server started. Bound to: 127.3.131.130:38319
I20260621 05:10:36.726296 7509 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.131.130:38319 every 8 connection(s)
I20260621 05:10:36.726629 7381 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782018616648578-3598-0/minicluster-data/ts-1/data/info.pb
I20260621 05:10:36.733207 3598 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWZOx2_/build/release/bin/kudu as pid 7381
I20260621 05:10:36.733335 3598 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782018616648578-3598-0/minicluster-data/ts-1/wal/instance
I20260621 05:10:36.733547 3598 external_mini_cluster.cc:1428] Setting key da1f42f57d1a4ca738e251334b6a2961
Jun 21 05:10:36 dist-test-slave-vrq1 krb5kdc[7140](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782018636, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.131.130@KRBTEST.COM for kudu/127.3.131.190@KRBTEST.COM
I20260621 05:10:36.738916 7224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:10:36.727488 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.131.130:58329 (local address 127.3.131.190:36241)
0621 05:10:36.727552 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0621 05:10:36.727555 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0621 05:10:36.727768 (+ 213us) server_negotiation.cc:407] Connection header received
0621 05:10:36.728735 (+ 967us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:10:36.728736 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:10:36.728747 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:10:36.729001 (+ 254us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:10:36.729552 (+ 551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:10:36.730054 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:10:36.731660 (+ 1606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:10:36.731870 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:10:36.731960 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:10:36.735014 (+ 3054us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:10:36.735016 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:10:36.735016 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:10:36.735026 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:10:36.737161 (+ 2135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:10:36.737672 (+ 511us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:10:36.737673 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:10:36.737673 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:10:36.737748 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:10:36.738137 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:10:36.738138 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:10:36.738138 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:10:36.738306 (+ 168us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:10:36.738380 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0621 05:10:36.738705 (+ 325us) server_negotiation.cc:299] Negotiation successful
0621 05:10:36.738799 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260621 05:10:36.739125 7512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:10:36.727482 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.131.190:36241 (local address 127.3.131.130:58329)
0621 05:10:36.727640 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0621 05:10:36.727677 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0621 05:10:36.728596 (+ 919us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:10:36.728952 (+ 356us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:10:36.728956 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:10:36.729224 (+ 268us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:10:36.729428 (+ 204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:10:36.729432 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:10:36.730193 (+ 761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:10:36.730194 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:10:36.731558 (+ 1364us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:10:36.731561 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:10:36.731970 (+ 409us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:10:36.731971 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:10:36.732128 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:10:36.732735 (+ 607us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 05:10:36.732748 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:10:36.734807 (+ 2059us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 05:10:36.737303 (+ 2496us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:10:36.737312 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:10:36.737315 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:10:36.737585 (+ 270us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:10:36.737901 (+ 316us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:10:36.737902 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:10:36.737903 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:10:36.738054 (+ 151us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:10:36.738396 (+ 342us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 05:10:36.738403 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 05:10:36.738546 (+ 143us) client_negotiation.cc:769] Sending connection context
0621 05:10:36.738664 (+ 118us) client_negotiation.cc:240] Negotiation successful
0621 05:10:36.738813 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":75,"threads_started":1}
I20260621 05:10:36.739858 7510 heartbeater.cc:347] Connected to a master server at 127.3.131.190:36241
I20260621 05:10:36.739981 7510 heartbeater.cc:464] Registering TS with master...
I20260621 05:10:36.740216 7510 heartbeater.cc:511] Master 127.3.131.190:36241 requested a full tablet report, sending...
I20260621 05:10:36.740669 7185 ts_manager.cc:194] Registered new tserver with Master: 76c09b10e2df47149979349cd84b2882 (127.3.131.130:38319)
I20260621 05:10:36.741044 7185 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.131.130@KRBTEST.COM'} at 127.3.131.130:58329
WARNING: no policy specified for kudu/127.3.131.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.131.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.131.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782018616648578-3598-0/krb5kdc/kudu_127.3.131.131.keytab.
Entry for principal kudu/127.3.131.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782018616648578-3598-0/krb5kdc/kudu_127.3.131.131.keytab.
I20260621 05:10:36.763126 3598 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWZOx2_/build/release/bin/kudu
/tmp/dist-test-taskWZOx2_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782018616648578-3598-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782018616648578-3598-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782018616648578-3598-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782018616648578-3598-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.131.131:0
--local_ip_for_outbound_sockets=127.3.131.131
--webserver_interface=127.3.131.131
--webserver_port=0
--tserver_master_addrs=127.3.131.190:36241
--keytab_file=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782018616648578-3598-0/krb5kdc/kudu_127.3.131.131.keytab
--principal=kudu/127.3.131.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.131.148:41969
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782018616648578-3598-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782018616648578-3598-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782018616648578-3598-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 05:10:36.857296 7515 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 05:10:36.857523 7515 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 05:10:36.857559 7515 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 05:10:36.858915 7515 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 05:10:36.858966 7515 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 05:10:36.858997 7515 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.131.131
I20260621 05:10:36.860688 7515 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.131.148:41969
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782018616648578-3598-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782018616648578-3598-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782018616648578-3598-0/krb5kdc/kudu_127.3.131.131.keytab
--principal=kudu/127.3.131.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.131.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782018616648578-3598-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.131.131
--webserver_port=0
--tserver_master_addrs=127.3.131.190:36241
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7515
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.131.131
--log_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782018616648578-3598-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 21 Jun 2026 04:16:16 UTC on ccycloud.quasar-xycaha.root.comops.site
build id 4457881
I20260621 05:10:36.860975 7515 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 05:10:36.861233 7515 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 05:10:36.866786 7523 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:10:36.866804 7521 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:10:36.866804 7520 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:10:36.867229 7515 server_base.cc:1034] running on GCE node
I20260621 05:10:36.867491 7515 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 05:10:36.867724 7515 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 05:10:36.868923 7515 hybrid_clock.cc:648] HybridClock initialized: now 1782018636868909 us; error 29 us; skew 500 ppm
Jun 21 05:10:36 dist-test-slave-vrq1 krb5kdc[7140](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782018636, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.131.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 05:10:36.872116 7515 init.cc:377] Logged in from keytab as kudu/127.3.131.131@KRBTEST.COM (short username kudu)
I20260621 05:10:36.872694 7515 webserver.cc:460] Webserver started at http://127.3.131.131:42317/ using document root <none> and password file <none>
I20260621 05:10:36.872954 7515 fs_manager.cc:359] Metadata directory not provided
I20260621 05:10:36.873008 7515 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 05:10:36.873165 7515 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 05:10:36.874471 7515 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782018616648578-3598-0/minicluster-data/ts-2/data/instance:
uuid: "0d3bf6900b6641cba95ec068bb436fec"
format_stamp: "Formatted at 2026-06-21 05:10:36 on dist-test-slave-vrq1"
server_key: "81dd7214f1e0c3e148ef781291413f31"
server_key_iv: "89d8153e0cd889166698b5abb57b754a"
server_key_version: "encryptionkey@0"
I20260621 05:10:36.874894 7515 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782018616648578-3598-0/minicluster-data/ts-2/wal/instance:
uuid: "0d3bf6900b6641cba95ec068bb436fec"
format_stamp: "Formatted at 2026-06-21 05:10:36 on dist-test-slave-vrq1"
server_key: "81dd7214f1e0c3e148ef781291413f31"
server_key_iv: "89d8153e0cd889166698b5abb57b754a"
server_key_version: "encryptionkey@0"
I20260621 05:10:36.876607 7515 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260621 05:10:36.877614 7530 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 05:10:36.877957 7515 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260621 05:10:36.878093 7515 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782018616648578-3598-0/minicluster-data/ts-2/data,/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782018616648578-3598-0/minicluster-data/ts-2/wal
uuid: "0d3bf6900b6641cba95ec068bb436fec"
format_stamp: "Formatted at 2026-06-21 05:10:36 on dist-test-slave-vrq1"
server_key: "81dd7214f1e0c3e148ef781291413f31"
server_key_iv: "89d8153e0cd889166698b5abb57b754a"
server_key_version: "encryptionkey@0"
I20260621 05:10:36.878191 7515 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782018616648578-3598-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782018616648578-3598-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782018616648578-3598-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 05:10:36.913930 7515 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 05:10:36.916319 7515 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 05:10:36.916509 7515 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 05:10:36.916775 7515 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 05:10:36.917093 7515 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 05:10:36.917150 7515 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 05:10:36.917192 7515 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 05:10:36.917249 7515 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 05:10:36.923048 7515 rpc_server.cc:307] RPC server started. Bound to: 127.3.131.131:46211
I20260621 05:10:36.923099 7643 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.131.131:46211 every 8 connection(s)
I20260621 05:10:36.923467 7515 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782018616648578-3598-0/minicluster-data/ts-2/data/info.pb
I20260621 05:10:36.929689 3598 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWZOx2_/build/release/bin/kudu as pid 7515
I20260621 05:10:36.929764 3598 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782018616648578-3598-0/minicluster-data/ts-2/wal/instance
I20260621 05:10:36.930050 3598 external_mini_cluster.cc:1428] Setting key abf7583edbcae9cb62c55238bb6b151b
Jun 21 05:10:36 dist-test-slave-vrq1 krb5kdc[7140](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782018636, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.131.131@KRBTEST.COM for kudu/127.3.131.190@KRBTEST.COM
I20260621 05:10:36.934470 7224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:10:36.924315 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.131.131:53277 (local address 127.3.131.190:36241)
0621 05:10:36.924409 (+ 94us) server_negotiation.cc:206] Beginning negotiation
0621 05:10:36.924412 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0621 05:10:36.924639 (+ 227us) server_negotiation.cc:407] Connection header received
0621 05:10:36.925603 (+ 964us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:10:36.925604 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:10:36.925633 (+ 29us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:10:36.925694 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:10:36.926377 (+ 683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:10:36.926910 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:10:36.928126 (+ 1216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:10:36.928332 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:10:36.928396 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:10:36.931141 (+ 2745us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:10:36.931143 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:10:36.931143 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:10:36.931152 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:10:36.932729 (+ 1577us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:10:36.933322 (+ 593us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:10:36.933324 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:10:36.933324 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:10:36.933400 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:10:36.933807 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:10:36.933808 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:10:36.933808 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:10:36.933935 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:10:36.933993 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0621 05:10:36.934307 (+ 314us) server_negotiation.cc:299] Negotiation successful
0621 05:10:36.934407 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260621 05:10:36.934621 7646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:10:36.924271 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.131.190:36241 (local address 127.3.131.131:53277)
0621 05:10:36.924456 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0621 05:10:36.924501 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0621 05:10:36.925472 (+ 971us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:10:36.925765 (+ 293us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:10:36.925769 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:10:36.926057 (+ 288us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:10:36.926265 (+ 208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:10:36.926269 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:10:36.927034 (+ 765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:10:36.927036 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:10:36.928009 (+ 973us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:10:36.928011 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:10:36.928437 (+ 426us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:10:36.928438 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:10:36.928564 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:10:36.929200 (+ 636us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 05:10:36.929209 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:10:36.931051 (+ 1842us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 05:10:36.932833 (+ 1782us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:10:36.932841 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:10:36.932843 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:10:36.933102 (+ 259us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:10:36.933533 (+ 431us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:10:36.933534 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:10:36.933535 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:10:36.933735 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:10:36.933985 (+ 250us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 05:10:36.933990 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 05:10:36.934163 (+ 173us) client_negotiation.cc:769] Sending connection context
0621 05:10:36.934253 (+ 90us) client_negotiation.cc:240] Negotiation successful
0621 05:10:36.934387 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":98,"threads_started":1}
I20260621 05:10:36.935215 7644 heartbeater.cc:347] Connected to a master server at 127.3.131.190:36241
I20260621 05:10:36.935316 7644 heartbeater.cc:464] Registering TS with master...
I20260621 05:10:36.935464 7644 heartbeater.cc:511] Master 127.3.131.190:36241 requested a full tablet report, sending...
I20260621 05:10:36.935878 7185 ts_manager.cc:194] Registered new tserver with Master: 0d3bf6900b6641cba95ec068bb436fec (127.3.131.131:46211)
I20260621 05:10:36.936218 7185 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.131.131@KRBTEST.COM'} at 127.3.131.131:53277
I20260621 05:10:36.942649 3598 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 21 05:10:36 dist-test-slave-vrq1 krb5kdc[7140](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782018636, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 21 05:10:36 dist-test-slave-vrq1 krb5kdc[7140](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782018636, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.131.129@KRBTEST.COM
I20260621 05:10:36.967269 7651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:10:36.958889 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47820 (local address 127.3.131.129:35745)
0621 05:10:36.959102 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0621 05:10:36.959106 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 05:10:36.959118 (+ 12us) server_negotiation.cc:407] Connection header received
0621 05:10:36.959151 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:10:36.959153 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:10:36.959250 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:10:36.959339 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:10:36.959996 (+ 657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:10:36.960769 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:10:36.961525 (+ 756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:10:36.961867 (+ 342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:10:36.961934 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:10:36.964328 (+ 2394us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:10:36.964335 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:10:36.964336 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:10:36.964343 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:10:36.965919 (+ 1576us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:10:36.966323 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:10:36.966327 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:10:36.966327 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:10:36.966386 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:10:36.966589 (+ 203us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:10:36.966590 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:10:36.966590 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:10:36.966738 (+ 148us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:10:36.966858 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0621 05:10:36.967005 (+ 147us) server_negotiation.cc:299] Negotiation successful
0621 05:10:36.967073 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":166,"thread_start_us":92,"threads_started":1}
Jun 21 05:10:36 dist-test-slave-vrq1 krb5kdc[7140](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782018636, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.131.190@KRBTEST.COM
I20260621 05:10:36.976503 7224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:10:36.968837 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55648 (local address 127.3.131.190:36241)
0621 05:10:36.968919 (+ 82us) server_negotiation.cc:206] Beginning negotiation
0621 05:10:36.968920 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0621 05:10:36.968929 (+ 9us) server_negotiation.cc:407] Connection header received
0621 05:10:36.968972 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:10:36.968973 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:10:36.968982 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:10:36.969031 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:10:36.969536 (+ 505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:10:36.970038 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:10:36.970863 (+ 825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:10:36.971049 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:10:36.971106 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:10:36.973650 (+ 2544us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:10:36.973652 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:10:36.973652 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:10:36.973659 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:10:36.975094 (+ 1435us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:10:36.975566 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:10:36.975567 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:10:36.975568 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:10:36.975633 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:10:36.975835 (+ 202us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:10:36.975836 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:10:36.975836 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:10:36.975990 (+ 154us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:10:36.976069 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0621 05:10:36.976345 (+ 276us) server_negotiation.cc:299] Negotiation successful
0621 05:10:36.976435 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
W20260621 05:10:36.976792 7185 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55648
I20260621 05:10:36.977240 3598 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWZOx2_/build/release/bin/kudu with pid 7247
I20260621 05:10:36.982028 3598 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWZOx2_/build/release/bin/kudu with pid 7381
I20260621 05:10:36.986624 3598 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWZOx2_/build/release/bin/kudu with pid 7515
I20260621 05:10:36.991072 3598 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWZOx2_/build/release/bin/kudu with pid 7153
2026-06-21T05:10:36Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (1980 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782018616648578-3598-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 21 05:10:37 dist-test-slave-vrq1 krb5kdc[7656](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 21 05:10:37 dist-test-slave-vrq1 krb5kdc[7656](info): set up 2 sockets
Jun 21 05:10:37 dist-test-slave-vrq1 krb5kdc[7656](info): commencing operation
krb5kdc: starting...
W20260621 05:10:38.051491 3598 mini_kdc.cc:121] Time spent starting KDC: real 1.039s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782018616648578-3598-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782018616648578-3598-0/krb5kdc/test-user.keytab.
Jun 21 05:10:38 dist-test-slave-vrq1 krb5kdc[7656](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782018638, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-21T05:10:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-21T05:10:38Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.131.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.131.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.131.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782018616648578-3598-0/krb5kdc/kudu_127.3.131.190.keytab.
Entry for principal kudu/127.3.131.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782018616648578-3598-0/krb5kdc/kudu_127.3.131.190.keytab.
I20260621 05:10:38.183555 3598 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWZOx2_/build/release/bin/kudu
/tmp/dist-test-taskWZOx2_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782018616648578-3598-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782018616648578-3598-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782018616648578-3598-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782018616648578-3598-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.131.190:35757
--webserver_interface=127.3.131.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782018616648578-3598-0/krb5kdc/kudu_127.3.131.190.keytab
--principal=kudu/127.3.131.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.131.148:43847
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.131.190:35757
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782018616648578-3598-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782018616648578-3598-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782018616648578-3598-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 05:10:38.268707 7669 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 05:10:38.268975 7669 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 05:10:38.269004 7669 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 05:10:38.270475 7669 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 05:10:38.270541 7669 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 05:10:38.270558 7669 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 05:10:38.270573 7669 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 05:10:38.270586 7669 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 05:10:38.272773 7669 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.131.148:43847
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782018616648578-3598-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782018616648578-3598-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.131.190:35757
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782018616648578-3598-0/krb5kdc/kudu_127.3.131.190.keytab
--principal=kudu/127.3.131.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.131.190:35757
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782018616648578-3598-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.131.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7669
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782018616648578-3598-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 21 Jun 2026 04:16:16 UTC on ccycloud.quasar-xycaha.root.comops.site
build id 4457881
I20260621 05:10:38.273115 7669 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 05:10:38.273419 7669 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 05:10:38.278626 7675 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:10:38.278628 7677 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:10:38.278687 7669 server_base.cc:1034] running on GCE node
W20260621 05:10:38.278626 7674 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:10:38.279275 7669 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 05:10:38.279598 7669 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 05:10:38.280790 7669 hybrid_clock.cc:648] HybridClock initialized: now 1782018638280783 us; error 37 us; skew 500 ppm
Jun 21 05:10:38 dist-test-slave-vrq1 krb5kdc[7656](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782018638, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.131.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 05:10:38.287492 7669 init.cc:377] Logged in from keytab as kudu/127.3.131.190@KRBTEST.COM (short username kudu)
I20260621 05:10:38.287894 7669 fs_manager.cc:359] Metadata directory not provided
I20260621 05:10:38.287976 7669 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 05:10:38.288141 7669 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 05:10:38.289304 7669 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782018616648578-3598-0/minicluster-data/master-0/data/instance:
uuid: "2f959bd270b8497a85a36fbdfea7ca32"
format_stamp: "Formatted at 2026-06-21 05:10:38 on dist-test-slave-vrq1"
server_key: "47e2250a089c5f0aea2a0af5b0603820"
server_key_iv: "6650f2013a9586b096190099528ea9ec"
server_key_version: "encryptionkey@0"
I20260621 05:10:38.289695 7669 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782018616648578-3598-0/minicluster-data/master-0/wal/instance:
uuid: "2f959bd270b8497a85a36fbdfea7ca32"
format_stamp: "Formatted at 2026-06-21 05:10:38 on dist-test-slave-vrq1"
server_key: "47e2250a089c5f0aea2a0af5b0603820"
server_key_iv: "6650f2013a9586b096190099528ea9ec"
server_key_version: "encryptionkey@0"
I20260621 05:10:38.291124 7669 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260621 05:10:38.292022 7683 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 05:10:38.292328 7669 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260621 05:10:38.292454 7669 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782018616648578-3598-0/minicluster-data/master-0/data,/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782018616648578-3598-0/minicluster-data/master-0/wal
uuid: "2f959bd270b8497a85a36fbdfea7ca32"
format_stamp: "Formatted at 2026-06-21 05:10:38 on dist-test-slave-vrq1"
server_key: "47e2250a089c5f0aea2a0af5b0603820"
server_key_iv: "6650f2013a9586b096190099528ea9ec"
server_key_version: "encryptionkey@0"
I20260621 05:10:38.292555 7669 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782018616648578-3598-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782018616648578-3598-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782018616648578-3598-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 05:10:38.301192 7669 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 05:10:38.303964 7669 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 05:10:38.304210 7669 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 05:10:38.308279 7669 rpc_server.cc:307] RPC server started. Bound to: 127.3.131.190:35757
I20260621 05:10:38.308303 7735 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.131.190:35757 every 8 connection(s)
I20260621 05:10:38.308614 7669 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782018616648578-3598-0/minicluster-data/master-0/data/info.pb
I20260621 05:10:38.308936 3598 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWZOx2_/build/release/bin/kudu as pid 7669
I20260621 05:10:38.309084 3598 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782018616648578-3598-0/minicluster-data/master-0/wal/instance
I20260621 05:10:38.309373 3598 external_mini_cluster.cc:1428] Setting key 6dc80f2022b67520c00020df9a4a120a
I20260621 05:10:38.309828 7736 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 05:10:38.313418 7736 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2f959bd270b8497a85a36fbdfea7ca32: Bootstrap starting.
I20260621 05:10:38.314113 7736 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2f959bd270b8497a85a36fbdfea7ca32: Neither blocks nor log segments found. Creating new log.
I20260621 05:10:38.314633 7736 log.cc:824] T 00000000000000000000000000000000 P 2f959bd270b8497a85a36fbdfea7ca32: Log is configured to *not* fsync() on all Append() calls
I20260621 05:10:38.315742 7736 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2f959bd270b8497a85a36fbdfea7ca32: No bootstrap required, opened a new log
Jun 21 05:10:38 dist-test-slave-vrq1 krb5kdc[7656](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782018638, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.131.190@KRBTEST.COM
I20260621 05:10:38.317916 7736 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2f959bd270b8497a85a36fbdfea7ca32 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f959bd270b8497a85a36fbdfea7ca32" member_type: VOTER last_known_addr { host: "127.3.131.190" port: 35757 } }
I20260621 05:10:38.318079 7736 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2f959bd270b8497a85a36fbdfea7ca32 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 05:10:38.318104 7736 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2f959bd270b8497a85a36fbdfea7ca32 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2f959bd270b8497a85a36fbdfea7ca32, State: Initialized, Role: FOLLOWER
I20260621 05:10:38.318272 7736 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2f959bd270b8497a85a36fbdfea7ca32 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f959bd270b8497a85a36fbdfea7ca32" member_type: VOTER last_known_addr { host: "127.3.131.190" port: 35757 } }
I20260621 05:10:38.318352 7736 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2f959bd270b8497a85a36fbdfea7ca32 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260621 05:10:38.318377 7736 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2f959bd270b8497a85a36fbdfea7ca32 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260621 05:10:38.318450 7736 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2f959bd270b8497a85a36fbdfea7ca32 [term 0 FOLLOWER]: Advancing to term 1
I20260621 05:10:38.319036 7736 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2f959bd270b8497a85a36fbdfea7ca32 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f959bd270b8497a85a36fbdfea7ca32" member_type: VOTER last_known_addr { host: "127.3.131.190" port: 35757 } }
I20260621 05:10:38.319124 7736 leader_election.cc:302] T 00000000000000000000000000000000 P 2f959bd270b8497a85a36fbdfea7ca32 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2f959bd270b8497a85a36fbdfea7ca32; no voters:
I20260621 05:10:38.319366 7736 leader_election.cc:288] T 00000000000000000000000000000000 P 2f959bd270b8497a85a36fbdfea7ca32 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260621 05:10:38.319399 7741 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2f959bd270b8497a85a36fbdfea7ca32 [term 1 FOLLOWER]: Leader election won for term 1
I20260621 05:10:38.319519 7741 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2f959bd270b8497a85a36fbdfea7ca32 [term 1 LEADER]: Becoming Leader. State: Replica: 2f959bd270b8497a85a36fbdfea7ca32, State: Running, Role: LEADER
I20260621 05:10:38.319613 7741 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2f959bd270b8497a85a36fbdfea7ca32 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f959bd270b8497a85a36fbdfea7ca32" member_type: VOTER last_known_addr { host: "127.3.131.190" port: 35757 } }
I20260621 05:10:38.319656 7736 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2f959bd270b8497a85a36fbdfea7ca32 [sys.catalog]: configured and running, proceeding with master startup.
I20260621 05:10:38.319932 7741 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2f959bd270b8497a85a36fbdfea7ca32 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2f959bd270b8497a85a36fbdfea7ca32" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f959bd270b8497a85a36fbdfea7ca32" member_type: VOTER last_known_addr { host: "127.3.131.190" port: 35757 } } }
I20260621 05:10:38.320010 7741 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2f959bd270b8497a85a36fbdfea7ca32 [sys.catalog]: This master's current role is: LEADER
I20260621 05:10:38.320142 7741 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2f959bd270b8497a85a36fbdfea7ca32 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2f959bd270b8497a85a36fbdfea7ca32. Latest consensus state: current_term: 1 leader_uuid: "2f959bd270b8497a85a36fbdfea7ca32" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f959bd270b8497a85a36fbdfea7ca32" member_type: VOTER last_known_addr { host: "127.3.131.190" port: 35757 } } }
I20260621 05:10:38.320209 7741 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2f959bd270b8497a85a36fbdfea7ca32 [sys.catalog]: This master's current role is: LEADER
I20260621 05:10:38.320607 7745 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260621 05:10:38.321786 7739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:10:38.309915 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47782 (local address 127.3.131.190:35757)
0621 05:10:38.310096 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0621 05:10:38.310150 (+ 54us) server_negotiation.cc:399] Waiting for connection header
0621 05:10:38.310178 (+ 28us) server_negotiation.cc:407] Connection header received
0621 05:10:38.310626 (+ 448us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:10:38.310633 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:10:38.310680 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:10:38.310793 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:10:38.311668 (+ 875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:10:38.312601 (+ 933us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:10:38.313693 (+ 1092us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:10:38.314126 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:10:38.314244 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:10:38.317241 (+ 2997us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:10:38.317248 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:10:38.317249 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:10:38.317266 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:10:38.319569 (+ 2303us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:10:38.320070 (+ 501us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:10:38.320073 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:10:38.320074 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:10:38.320198 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:10:38.320476 (+ 278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:10:38.320486 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:10:38.320486 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:10:38.320880 (+ 394us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:10:38.321057 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0621 05:10:38.321428 (+ 371us) server_negotiation.cc:299] Negotiation successful
0621 05:10:38.321569 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":126,"thread_start_us":90,"threads_started":1}
I20260621 05:10:38.322085 7745 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260621 05:10:38.323818 7745 catalog_manager.cc:1347] Generated new cluster ID: 8b81bf304bc549fe801d3f723915ce05
I20260621 05:10:38.323905 7745 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260621 05:10:38.330806 7745 catalog_manager.cc:1370] Generated new certificate authority record
I20260621 05:10:38.331575 7745 catalog_manager.cc:1504] Loading token signing keys...
I20260621 05:10:38.336942 7745 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2f959bd270b8497a85a36fbdfea7ca32: Generated new TSK 0
I20260621 05:10:38.337214 7745 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.3.131.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.131.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.131.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782018616648578-3598-0/krb5kdc/kudu_127.3.131.129.keytab.
Entry for principal kudu/127.3.131.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782018616648578-3598-0/krb5kdc/kudu_127.3.131.129.keytab.
I20260621 05:10:38.374799 3598 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWZOx2_/build/release/bin/kudu
/tmp/dist-test-taskWZOx2_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782018616648578-3598-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782018616648578-3598-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782018616648578-3598-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782018616648578-3598-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.131.129:0
--local_ip_for_outbound_sockets=127.3.131.129
--webserver_interface=127.3.131.129
--webserver_port=0
--tserver_master_addrs=127.3.131.190:35757
--keytab_file=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782018616648578-3598-0/krb5kdc/kudu_127.3.131.129.keytab
--principal=kudu/127.3.131.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.131.148:43847
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782018616648578-3598-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782018616648578-3598-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782018616648578-3598-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 05:10:38.478533 7762 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 05:10:38.478753 7762 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 05:10:38.478780 7762 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 05:10:38.480048 7762 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 05:10:38.480099 7762 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 05:10:38.480129 7762 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.131.129
I20260621 05:10:38.481688 7762 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.131.148:43847
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782018616648578-3598-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782018616648578-3598-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782018616648578-3598-0/krb5kdc/kudu_127.3.131.129.keytab
--principal=kudu/127.3.131.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.131.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782018616648578-3598-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.131.129
--webserver_port=0
--tserver_master_addrs=127.3.131.190:35757
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7762
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.131.129
--log_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782018616648578-3598-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 21 Jun 2026 04:16:16 UTC on ccycloud.quasar-xycaha.root.comops.site
build id 4457881
I20260621 05:10:38.481953 7762 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 05:10:38.482203 7762 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 05:10:38.487358 7767 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:10:38.487627 7770 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:10:38.487696 7768 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:10:38.487671 7762 server_base.cc:1034] running on GCE node
I20260621 05:10:38.488039 7762 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 05:10:38.488399 7762 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 05:10:38.489562 7762 hybrid_clock.cc:648] HybridClock initialized: now 1782018638489531 us; error 44 us; skew 500 ppm
Jun 21 05:10:38 dist-test-slave-vrq1 krb5kdc[7656](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782018638, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.131.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 05:10:38.492713 7762 init.cc:377] Logged in from keytab as kudu/127.3.131.129@KRBTEST.COM (short username kudu)
I20260621 05:10:38.493072 7762 fs_manager.cc:359] Metadata directory not provided
I20260621 05:10:38.493202 7762 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 05:10:38.493366 7762 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 05:10:38.494560 7762 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782018616648578-3598-0/minicluster-data/ts-0/data/instance:
uuid: "4d3490def2e54588b8d71a7493761370"
format_stamp: "Formatted at 2026-06-21 05:10:38 on dist-test-slave-vrq1"
server_key: "b371dc5663319e01c21f0df995f869ae"
server_key_iv: "5dbaf801830c5d454cfd883f645e3cd7"
server_key_version: "encryptionkey@0"
I20260621 05:10:38.494937 7762 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782018616648578-3598-0/minicluster-data/ts-0/wal/instance:
uuid: "4d3490def2e54588b8d71a7493761370"
format_stamp: "Formatted at 2026-06-21 05:10:38 on dist-test-slave-vrq1"
server_key: "b371dc5663319e01c21f0df995f869ae"
server_key_iv: "5dbaf801830c5d454cfd883f645e3cd7"
server_key_version: "encryptionkey@0"
I20260621 05:10:38.496464 7762 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260621 05:10:38.497469 7776 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 05:10:38.497756 7762 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260621 05:10:38.497872 7762 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782018616648578-3598-0/minicluster-data/ts-0/data,/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782018616648578-3598-0/minicluster-data/ts-0/wal
uuid: "4d3490def2e54588b8d71a7493761370"
format_stamp: "Formatted at 2026-06-21 05:10:38 on dist-test-slave-vrq1"
server_key: "b371dc5663319e01c21f0df995f869ae"
server_key_iv: "5dbaf801830c5d454cfd883f645e3cd7"
server_key_version: "encryptionkey@0"
I20260621 05:10:38.497951 7762 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782018616648578-3598-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782018616648578-3598-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782018616648578-3598-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 05:10:38.530157 7762 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 05:10:38.532714 7762 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 05:10:38.532892 7762 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 05:10:38.533154 7762 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 05:10:38.533484 7762 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 05:10:38.533545 7762 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 05:10:38.533597 7762 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 05:10:38.533627 7762 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 05:10:38.539583 7762 rpc_server.cc:307] RPC server started. Bound to: 127.3.131.129:33077
I20260621 05:10:38.539685 7889 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.131.129:33077 every 8 connection(s)
I20260621 05:10:38.539969 7762 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782018616648578-3598-0/minicluster-data/ts-0/data/info.pb
I20260621 05:10:38.541515 3598 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWZOx2_/build/release/bin/kudu as pid 7762
I20260621 05:10:38.541632 3598 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782018616648578-3598-0/minicluster-data/ts-0/wal/instance
I20260621 05:10:38.541844 3598 external_mini_cluster.cc:1428] Setting key 995bf67c491bb42be83527d3bfd24384
Jun 21 05:10:38 dist-test-slave-vrq1 krb5kdc[7656](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782018638, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.131.129@KRBTEST.COM for kudu/127.3.131.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.3.131.130@KRBTEST.COM; defaulting to no policy
I20260621 05:10:38.551798 7739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:10:38.540829 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.131.129:35619 (local address 127.3.131.190:35757)
0621 05:10:38.540921 (+ 92us) server_negotiation.cc:206] Beginning negotiation
0621 05:10:38.540923 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:10:38.541122 (+ 199us) server_negotiation.cc:407] Connection header received
0621 05:10:38.541734 (+ 612us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:10:38.541736 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:10:38.541751 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:10:38.541824 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:10:38.543012 (+ 1188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:10:38.543823 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:10:38.545462 (+ 1639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:10:38.545700 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:10:38.545768 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:10:38.548403 (+ 2635us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:10:38.548405 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:10:38.548405 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:10:38.548413 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:10:38.550049 (+ 1636us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:10:38.550536 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:10:38.550537 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:10:38.550537 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:10:38.550604 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:10:38.550944 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:10:38.550945 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:10:38.550945 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:10:38.551069 (+ 124us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:10:38.551234 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0621 05:10:38.551510 (+ 276us) server_negotiation.cc:299] Negotiation successful
0621 05:10:38.551667 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260621 05:10:38.551811 7892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:10:38.540801 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.131.190:35757 (local address 127.3.131.129:35619)
0621 05:10:38.540987 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0621 05:10:38.541031 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0621 05:10:38.541598 (+ 567us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:10:38.541879 (+ 281us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:10:38.541882 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:10:38.542293 (+ 411us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:10:38.542903 (+ 610us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:10:38.542909 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:10:38.543952 (+ 1043us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:10:38.543953 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:10:38.545355 (+ 1402us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:10:38.545358 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:10:38.545796 (+ 438us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:10:38.545797 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:10:38.545955 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:10:38.546561 (+ 606us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 05:10:38.546573 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:10:38.548296 (+ 1723us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 05:10:38.550209 (+ 1913us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:10:38.550217 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:10:38.550219 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:10:38.550454 (+ 235us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:10:38.550682 (+ 228us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:10:38.550684 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:10:38.550684 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:10:38.550838 (+ 154us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:10:38.551214 (+ 376us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 05:10:38.551221 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 05:10:38.551359 (+ 138us) client_negotiation.cc:769] Sending connection context
0621 05:10:38.551451 (+ 92us) client_negotiation.cc:240] Negotiation successful
0621 05:10:38.551580 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":91,"threads_started":1}
I20260621 05:10:38.552405 7890 heartbeater.cc:347] Connected to a master server at 127.3.131.190:35757
I20260621 05:10:38.552525 7890 heartbeater.cc:464] Registering TS with master...
I20260621 05:10:38.552706 7890 heartbeater.cc:511] Master 127.3.131.190:35757 requested a full tablet report, sending...
I20260621 05:10:38.553212 7699 ts_manager.cc:194] Registered new tserver with Master: 4d3490def2e54588b8d71a7493761370 (127.3.131.129:33077)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.131.130@KRBTEST.COM" created.
I20260621 05:10:38.554069 7699 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.131.129@KRBTEST.COM'} at 127.3.131.129:35619
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.131.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782018616648578-3598-0/krb5kdc/kudu_127.3.131.130.keytab.
Entry for principal kudu/127.3.131.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782018616648578-3598-0/krb5kdc/kudu_127.3.131.130.keytab.
I20260621 05:10:38.571318 3598 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWZOx2_/build/release/bin/kudu
/tmp/dist-test-taskWZOx2_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782018616648578-3598-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782018616648578-3598-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782018616648578-3598-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782018616648578-3598-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.131.130:0
--local_ip_for_outbound_sockets=127.3.131.130
--webserver_interface=127.3.131.130
--webserver_port=0
--tserver_master_addrs=127.3.131.190:35757
--keytab_file=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782018616648578-3598-0/krb5kdc/kudu_127.3.131.130.keytab
--principal=kudu/127.3.131.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.131.148:43847
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782018616648578-3598-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782018616648578-3598-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782018616648578-3598-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 05:10:38.658346 7895 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 05:10:38.658569 7895 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 05:10:38.658595 7895 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 05:10:38.659843 7895 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 05:10:38.659897 7895 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 05:10:38.659925 7895 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.131.130
I20260621 05:10:38.661466 7895 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.131.148:43847
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782018616648578-3598-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782018616648578-3598-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782018616648578-3598-0/krb5kdc/kudu_127.3.131.130.keytab
--principal=kudu/127.3.131.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.131.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782018616648578-3598-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.131.130
--webserver_port=0
--tserver_master_addrs=127.3.131.190:35757
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7895
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.131.130
--log_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782018616648578-3598-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 21 Jun 2026 04:16:16 UTC on ccycloud.quasar-xycaha.root.comops.site
build id 4457881
I20260621 05:10:38.661720 7895 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 05:10:38.661998 7895 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 05:10:38.667304 7901 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:10:38.667416 7895 server_base.cc:1034] running on GCE node
W20260621 05:10:38.667304 7900 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:10:38.667300 7903 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:10:38.667773 7895 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 05:10:38.668026 7895 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 05:10:38.669198 7895 hybrid_clock.cc:648] HybridClock initialized: now 1782018638669185 us; error 49 us; skew 500 ppm
Jun 21 05:10:38 dist-test-slave-vrq1 krb5kdc[7656](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782018638, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.131.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 05:10:38.672145 7895 init.cc:377] Logged in from keytab as kudu/127.3.131.130@KRBTEST.COM (short username kudu)
I20260621 05:10:38.672556 7895 fs_manager.cc:359] Metadata directory not provided
I20260621 05:10:38.672640 7895 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 05:10:38.672802 7895 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 05:10:38.673908 7895 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782018616648578-3598-0/minicluster-data/ts-1/data/instance:
uuid: "f47decfebb6c4a78a0f7a9126f201e08"
format_stamp: "Formatted at 2026-06-21 05:10:38 on dist-test-slave-vrq1"
server_key: "cdd02aaa3e83d8d9908b0d521802f88c"
server_key_iv: "5be7ec6b345ca735c175243ab8e6d851"
server_key_version: "encryptionkey@0"
I20260621 05:10:38.674291 7895 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782018616648578-3598-0/minicluster-data/ts-1/wal/instance:
uuid: "f47decfebb6c4a78a0f7a9126f201e08"
format_stamp: "Formatted at 2026-06-21 05:10:38 on dist-test-slave-vrq1"
server_key: "cdd02aaa3e83d8d9908b0d521802f88c"
server_key_iv: "5be7ec6b345ca735c175243ab8e6d851"
server_key_version: "encryptionkey@0"
I20260621 05:10:38.675776 7895 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260621 05:10:38.676630 7909 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 05:10:38.676810 7895 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260621 05:10:38.676910 7895 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782018616648578-3598-0/minicluster-data/ts-1/data,/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782018616648578-3598-0/minicluster-data/ts-1/wal
uuid: "f47decfebb6c4a78a0f7a9126f201e08"
format_stamp: "Formatted at 2026-06-21 05:10:38 on dist-test-slave-vrq1"
server_key: "cdd02aaa3e83d8d9908b0d521802f88c"
server_key_iv: "5be7ec6b345ca735c175243ab8e6d851"
server_key_version: "encryptionkey@0"
I20260621 05:10:38.676999 7895 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782018616648578-3598-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782018616648578-3598-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782018616648578-3598-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 05:10:38.699054 7895 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 05:10:38.701808 7895 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 05:10:38.702075 7895 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 05:10:38.702358 7895 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 05:10:38.702710 7895 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 05:10:38.702764 7895 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 05:10:38.702814 7895 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 05:10:38.702863 7895 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 05:10:38.709581 7895 rpc_server.cc:307] RPC server started. Bound to: 127.3.131.130:43689
I20260621 05:10:38.709625 8022 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.131.130:43689 every 8 connection(s)
I20260621 05:10:38.709920 7895 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782018616648578-3598-0/minicluster-data/ts-1/data/info.pb
Jun 21 05:10:38 dist-test-slave-vrq1 krb5kdc[7656](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782018638, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.131.130@KRBTEST.COM for kudu/127.3.131.190@KRBTEST.COM
I20260621 05:10:38.717250 3598 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWZOx2_/build/release/bin/kudu as pid 7895
I20260621 05:10:38.717386 3598 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782018616648578-3598-0/minicluster-data/ts-1/wal/instance
I20260621 05:10:38.717682 3598 external_mini_cluster.cc:1428] Setting key e7fa008014a9f2f3baa127783228d2a6
I20260621 05:10:38.721144 7739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:10:38.710755 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.131.130:49427 (local address 127.3.131.190:35757)
0621 05:10:38.710827 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0621 05:10:38.710830 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0621 05:10:38.711025 (+ 195us) server_negotiation.cc:407] Connection header received
0621 05:10:38.711533 (+ 508us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:10:38.711534 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:10:38.711545 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:10:38.711703 (+ 158us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:10:38.712720 (+ 1017us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:10:38.713219 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:10:38.714468 (+ 1249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:10:38.714667 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:10:38.714738 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:10:38.717321 (+ 2583us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:10:38.717324 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:10:38.717324 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:10:38.717335 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:10:38.718945 (+ 1610us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:10:38.719602 (+ 657us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:10:38.719603 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:10:38.719604 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:10:38.719810 (+ 206us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:10:38.720219 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:10:38.720221 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:10:38.720221 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:10:38.720445 (+ 224us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:10:38.720538 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0621 05:10:38.720906 (+ 368us) server_negotiation.cc:299] Negotiation successful
0621 05:10:38.721002 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260621 05:10:38.721282 8025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:10:38.710663 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.131.190:35757 (local address 127.3.131.130:49427)
0621 05:10:38.710841 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0621 05:10:38.710889 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0621 05:10:38.711436 (+ 547us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:10:38.711792 (+ 356us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:10:38.711796 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:10:38.712100 (+ 304us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:10:38.712617 (+ 517us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:10:38.712622 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:10:38.713381 (+ 759us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:10:38.713382 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:10:38.714371 (+ 989us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:10:38.714374 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:10:38.714798 (+ 424us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:10:38.714799 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:10:38.714919 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:10:38.715607 (+ 688us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 05:10:38.715617 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:10:38.717210 (+ 1593us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 05:10:38.719049 (+ 1839us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:10:38.719057 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:10:38.719060 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:10:38.719506 (+ 446us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:10:38.719943 (+ 437us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:10:38.719944 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:10:38.719945 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:10:38.720123 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:10:38.720532 (+ 409us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 05:10:38.720539 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 05:10:38.720725 (+ 186us) client_negotiation.cc:769] Sending connection context
0621 05:10:38.720821 (+ 96us) client_negotiation.cc:240] Negotiation successful
0621 05:10:38.721050 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":104,"threads_started":1}
I20260621 05:10:38.721865 8023 heartbeater.cc:347] Connected to a master server at 127.3.131.190:35757
I20260621 05:10:38.721978 8023 heartbeater.cc:464] Registering TS with master...
I20260621 05:10:38.722141 8023 heartbeater.cc:511] Master 127.3.131.190:35757 requested a full tablet report, sending...
I20260621 05:10:38.722581 7699 ts_manager.cc:194] Registered new tserver with Master: f47decfebb6c4a78a0f7a9126f201e08 (127.3.131.130:43689)
I20260621 05:10:38.722985 7699 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.131.130@KRBTEST.COM'} at 127.3.131.130:49427
WARNING: no policy specified for kudu/127.3.131.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.131.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.131.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782018616648578-3598-0/krb5kdc/kudu_127.3.131.131.keytab.
Entry for principal kudu/127.3.131.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782018616648578-3598-0/krb5kdc/kudu_127.3.131.131.keytab.
I20260621 05:10:38.747557 3598 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWZOx2_/build/release/bin/kudu
/tmp/dist-test-taskWZOx2_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782018616648578-3598-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782018616648578-3598-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782018616648578-3598-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782018616648578-3598-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.131.131:0
--local_ip_for_outbound_sockets=127.3.131.131
--webserver_interface=127.3.131.131
--webserver_port=0
--tserver_master_addrs=127.3.131.190:35757
--keytab_file=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782018616648578-3598-0/krb5kdc/kudu_127.3.131.131.keytab
--principal=kudu/127.3.131.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.131.148:43847
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782018616648578-3598-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782018616648578-3598-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782018616648578-3598-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 05:10:38.832755 8028 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 05:10:38.832983 8028 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 05:10:38.833009 8028 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 05:10:38.834194 8028 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 05:10:38.834233 8028 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 05:10:38.834261 8028 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.131.131
I20260621 05:10:38.836136 8028 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.131.148:43847
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782018616648578-3598-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782018616648578-3598-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782018616648578-3598-0/krb5kdc/kudu_127.3.131.131.keytab
--principal=kudu/127.3.131.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.131.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782018616648578-3598-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.131.131
--webserver_port=0
--tserver_master_addrs=127.3.131.190:35757
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8028
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.131.131
--log_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782018616648578-3598-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 21 Jun 2026 04:16:16 UTC on ccycloud.quasar-xycaha.root.comops.site
build id 4457881
I20260621 05:10:38.836387 8028 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 05:10:38.836575 8028 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 05:10:38.842049 8034 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:10:38.842121 8033 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:10:38.842118 8036 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:10:38.842162 8028 server_base.cc:1034] running on GCE node
I20260621 05:10:38.842512 8028 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 05:10:38.842721 8028 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 05:10:38.843868 8028 hybrid_clock.cc:648] HybridClock initialized: now 1782018638843855 us; error 28 us; skew 500 ppm
Jun 21 05:10:38 dist-test-slave-vrq1 krb5kdc[7656](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782018638, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.131.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 05:10:38.846475 8028 init.cc:377] Logged in from keytab as kudu/127.3.131.131@KRBTEST.COM (short username kudu)
I20260621 05:10:38.846848 8028 fs_manager.cc:359] Metadata directory not provided
I20260621 05:10:38.846897 8028 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 05:10:38.846985 8028 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 05:10:38.848029 8028 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782018616648578-3598-0/minicluster-data/ts-2/data/instance:
uuid: "7e1d10dd516045ad8959b889d7ac422e"
format_stamp: "Formatted at 2026-06-21 05:10:38 on dist-test-slave-vrq1"
server_key: "49fb8020c0413c1f83c7f42079f43756"
server_key_iv: "42d8e8e52b792a9b13af7096949ba9f5"
server_key_version: "encryptionkey@0"
I20260621 05:10:38.848361 8028 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782018616648578-3598-0/minicluster-data/ts-2/wal/instance:
uuid: "7e1d10dd516045ad8959b889d7ac422e"
format_stamp: "Formatted at 2026-06-21 05:10:38 on dist-test-slave-vrq1"
server_key: "49fb8020c0413c1f83c7f42079f43756"
server_key_iv: "42d8e8e52b792a9b13af7096949ba9f5"
server_key_version: "encryptionkey@0"
I20260621 05:10:38.849582 8028 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260621 05:10:38.850303 8042 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 05:10:38.850477 8028 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260621 05:10:38.850581 8028 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782018616648578-3598-0/minicluster-data/ts-2/data,/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782018616648578-3598-0/minicluster-data/ts-2/wal
uuid: "7e1d10dd516045ad8959b889d7ac422e"
format_stamp: "Formatted at 2026-06-21 05:10:38 on dist-test-slave-vrq1"
server_key: "49fb8020c0413c1f83c7f42079f43756"
server_key_iv: "42d8e8e52b792a9b13af7096949ba9f5"
server_key_version: "encryptionkey@0"
I20260621 05:10:38.850656 8028 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782018616648578-3598-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782018616648578-3598-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782018616648578-3598-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 05:10:38.866189 8028 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 05:10:38.868320 8028 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 05:10:38.868528 8028 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 05:10:38.868769 8028 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 05:10:38.869097 8028 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 05:10:38.869153 8028 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 05:10:38.869197 8028 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 05:10:38.869222 8028 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 05:10:38.875597 8028 rpc_server.cc:307] RPC server started. Bound to: 127.3.131.131:38381
I20260621 05:10:38.875643 8155 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.131.131:38381 every 8 connection(s)
I20260621 05:10:38.875962 8028 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782018616648578-3598-0/minicluster-data/ts-2/data/info.pb
Jun 21 05:10:38 dist-test-slave-vrq1 krb5kdc[7656](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782018638, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.131.131@KRBTEST.COM for kudu/127.3.131.190@KRBTEST.COM
I20260621 05:10:38.883051 3598 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWZOx2_/build/release/bin/kudu as pid 8028
I20260621 05:10:38.883164 3598 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782018616648578-3598-0/minicluster-data/ts-2/wal/instance
I20260621 05:10:38.883389 3598 external_mini_cluster.cc:1428] Setting key 63d1aa0aea6b1635a9edde0a53de1d7c
I20260621 05:10:38.886511 7739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:10:38.876990 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.131.131:37255 (local address 127.3.131.190:35757)
0621 05:10:38.877060 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0621 05:10:38.877062 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:10:38.877348 (+ 286us) server_negotiation.cc:407] Connection header received
0621 05:10:38.877790 (+ 442us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:10:38.877792 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:10:38.877802 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:10:38.877874 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:10:38.878838 (+ 964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:10:38.879354 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:10:38.880489 (+ 1135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:10:38.880697 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:10:38.880764 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:10:38.883241 (+ 2477us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:10:38.883243 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:10:38.883243 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:10:38.883252 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:10:38.885061 (+ 1809us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:10:38.885526 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:10:38.885527 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:10:38.885528 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:10:38.885588 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:10:38.885820 (+ 232us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:10:38.885821 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:10:38.885821 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:10:38.885940 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:10:38.885997 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0621 05:10:38.886363 (+ 366us) server_negotiation.cc:299] Negotiation successful
0621 05:10:38.886437 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260621 05:10:38.886641 8158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:10:38.876973 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.131.190:35757 (local address 127.3.131.131:37255)
0621 05:10:38.877145 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0621 05:10:38.877189 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0621 05:10:38.877689 (+ 500us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:10:38.877934 (+ 245us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:10:38.877937 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:10:38.878223 (+ 286us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:10:38.878732 (+ 509us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:10:38.878738 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:10:38.879441 (+ 703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:10:38.879442 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:10:38.880389 (+ 947us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:10:38.880392 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:10:38.880790 (+ 398us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:10:38.880791 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:10:38.880894 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:10:38.881495 (+ 601us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 05:10:38.881505 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:10:38.883046 (+ 1541us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 05:10:38.885164 (+ 2118us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:10:38.885171 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:10:38.885174 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:10:38.885408 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:10:38.885656 (+ 248us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:10:38.885657 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:10:38.885657 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:10:38.885759 (+ 102us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:10:38.886011 (+ 252us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 05:10:38.886020 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 05:10:38.886196 (+ 176us) client_negotiation.cc:769] Sending connection context
0621 05:10:38.886288 (+ 92us) client_negotiation.cc:240] Negotiation successful
0621 05:10:38.886419 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":91,"threads_started":1}
I20260621 05:10:38.887111 8156 heartbeater.cc:347] Connected to a master server at 127.3.131.190:35757
I20260621 05:10:38.887253 8156 heartbeater.cc:464] Registering TS with master...
I20260621 05:10:38.887411 8156 heartbeater.cc:511] Master 127.3.131.190:35757 requested a full tablet report, sending...
I20260621 05:10:38.887820 7699 ts_manager.cc:194] Registered new tserver with Master: 7e1d10dd516045ad8959b889d7ac422e (127.3.131.131:38381)
I20260621 05:10:38.888207 7699 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.131.131@KRBTEST.COM'} at 127.3.131.131:37255
I20260621 05:10:38.896030 3598 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260621 05:10:38.904580 7739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:10:38.897481 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47796 (local address 127.3.131.190:35757)
0621 05:10:38.897571 (+ 90us) server_negotiation.cc:206] Beginning negotiation
0621 05:10:38.897573 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:10:38.897582 (+ 9us) server_negotiation.cc:407] Connection header received
0621 05:10:38.897622 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:10:38.897623 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:10:38.897632 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:10:38.897693 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:10:38.898379 (+ 686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:10:38.898880 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:10:38.899880 (+ 1000us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:10:38.900144 (+ 264us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:10:38.900229 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:10:38.901715 (+ 1486us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:10:38.901717 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:10:38.901717 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:10:38.901724 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:10:38.903311 (+ 1587us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:10:38.903728 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:10:38.903729 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:10:38.903729 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:10:38.903798 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:10:38.904030 (+ 232us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:10:38.904031 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:10:38.904031 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:10:38.904234 (+ 203us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:10:38.904321 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0621 05:10:38.904434 (+ 113us) server_negotiation.cc:299] Negotiation successful
0621 05:10:38.904502 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260621 05:10:38.906560 7699 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47796:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260621 05:10:38.908445 7699 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260621 05:10:38.915469 8168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:10:38.911346 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.131.130:43689 (local address 127.0.0.1:59756)
0621 05:10:38.911719 (+ 373us) negotiation.cc:107] Waiting for socket to connect
0621 05:10:38.911731 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0621 05:10:38.911798 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:10:38.912085 (+ 287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:10:38.912089 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:10:38.912091 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 05:10:38.912167 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:10:38.912170 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:10:38.913091 (+ 921us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:10:38.913093 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:10:38.914286 (+ 1193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:10:38.914289 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:10:38.915136 (+ 847us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:10:38.915137 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:10:38.915236 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:10:38.915252 (+ 16us) client_negotiation.cc:769] Sending connection context
0621 05:10:38.915328 (+ 76us) client_negotiation.cc:240] Negotiation successful
0621 05:10:38.915384 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":351,"mutex_wait_us":4,"spinlock_wait_cycles":11136,"thread_start_us":100,"threads_started":1}
I20260621 05:10:38.915479 8169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:10:38.911530 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59756 (local address 127.3.131.130:43689)
0621 05:10:38.911734 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0621 05:10:38.911739 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 05:10:38.911862 (+ 123us) server_negotiation.cc:407] Connection header received
0621 05:10:38.911900 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:10:38.911903 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:10:38.911920 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:10:38.911986 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 05:10:38.912254 (+ 268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:10:38.912986 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:10:38.914371 (+ 1385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:10:38.915049 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:10:38.915138 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:10:38.915239 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0621 05:10:38.915342 (+ 103us) server_negotiation.cc:299] Negotiation successful
0621 05:10:38.915385 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":167,"thread_start_us":106,"threads_started":1}
I20260621 05:10:38.915838 8171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:10:38.911549 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.131.131:38381 (local address 127.0.0.1:42058)
0621 05:10:38.911793 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0621 05:10:38.911802 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0621 05:10:38.911846 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:10:38.912185 (+ 339us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:10:38.912186 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:10:38.912187 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 05:10:38.912238 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:10:38.912239 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:10:38.913581 (+ 1342us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:10:38.913582 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:10:38.914600 (+ 1018us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:10:38.914602 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:10:38.915619 (+ 1017us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:10:38.915620 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:10:38.915691 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:10:38.915696 (+ 5us) client_negotiation.cc:769] Sending connection context
0621 05:10:38.915757 (+ 61us) client_negotiation.cc:240] Negotiation successful
0621 05:10:38.915789 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":226,"mutex_wait_us":2,"thread_start_us":81,"threads_started":1}
I20260621 05:10:38.915918 8172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:10:38.911771 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42058 (local address 127.3.131.131:38381)
0621 05:10:38.912015 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0621 05:10:38.912020 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 05:10:38.912029 (+ 9us) server_negotiation.cc:407] Connection header received
0621 05:10:38.912070 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:10:38.912073 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:10:38.912090 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:10:38.912316 (+ 226us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 05:10:38.912396 (+ 80us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:10:38.913479 (+ 1083us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:10:38.914696 (+ 1217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:10:38.915501 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:10:38.915615 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:10:38.915686 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0621 05:10:38.915785 (+ 99us) server_negotiation.cc:299] Negotiation successful
0621 05:10:38.915827 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":211,"thread_start_us":105,"threads_started":1}
I20260621 05:10:38.916417 7957 tablet_service.cc:1467] Processing CreateTablet for tablet 9c241e1876e04e84af26690f22910c79 (DEFAULT_TABLE table=test-table [id=f6291787ef82483991f83f2c184d0b48]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 05:10:38.916545 8167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:10:38.911346 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.131.129:33077 (local address 127.0.0.1:49208)
0621 05:10:38.911521 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0621 05:10:38.911565 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0621 05:10:38.911633 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:10:38.912090 (+ 457us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:10:38.912091 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:10:38.912093 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 05:10:38.912180 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:10:38.912181 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:10:38.913524 (+ 1343us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:10:38.913525 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:10:38.914708 (+ 1183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:10:38.914711 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:10:38.916302 (+ 1591us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:10:38.916304 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:10:38.916367 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:10:38.916376 (+ 9us) client_negotiation.cc:769] Sending connection context
0621 05:10:38.916431 (+ 55us) client_negotiation.cc:240] Negotiation successful
0621 05:10:38.916475 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":149,"mutex_wait_us":56,"thread_start_us":63,"threads_started":1}
I20260621 05:10:38.916558 8170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:10:38.911529 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49208 (local address 127.3.131.129:33077)
0621 05:10:38.911806 (+ 277us) server_negotiation.cc:206] Beginning negotiation
0621 05:10:38.911810 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 05:10:38.911820 (+ 10us) server_negotiation.cc:407] Connection header received
0621 05:10:38.911867 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:10:38.911869 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:10:38.911887 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:10:38.911978 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 05:10:38.912265 (+ 287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:10:38.913387 (+ 1122us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:10:38.914793 (+ 1406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:10:38.915527 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:10:38.916301 (+ 774us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:10:38.916367 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0621 05:10:38.916442 (+ 75us) server_negotiation.cc:299] Negotiation successful
0621 05:10:38.916483 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":248,"thread_start_us":91,"threads_started":1}
I20260621 05:10:38.916824 7957 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9c241e1876e04e84af26690f22910c79. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 05:10:38.919667 8173 tablet_bootstrap.cc:492] T 9c241e1876e04e84af26690f22910c79 P f47decfebb6c4a78a0f7a9126f201e08: Bootstrap starting.
I20260621 05:10:38.919770 8090 tablet_service.cc:1467] Processing CreateTablet for tablet 9c241e1876e04e84af26690f22910c79 (DEFAULT_TABLE table=test-table [id=f6291787ef82483991f83f2c184d0b48]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 05:10:38.919821 7824 tablet_service.cc:1467] Processing CreateTablet for tablet 9c241e1876e04e84af26690f22910c79 (DEFAULT_TABLE table=test-table [id=f6291787ef82483991f83f2c184d0b48]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 05:10:38.920135 8090 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9c241e1876e04e84af26690f22910c79. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 05:10:38.920222 7824 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9c241e1876e04e84af26690f22910c79. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 05:10:38.920397 8173 tablet_bootstrap.cc:654] T 9c241e1876e04e84af26690f22910c79 P f47decfebb6c4a78a0f7a9126f201e08: Neither blocks nor log segments found. Creating new log.
I20260621 05:10:38.921007 8173 log.cc:824] T 9c241e1876e04e84af26690f22910c79 P f47decfebb6c4a78a0f7a9126f201e08: Log is configured to *not* fsync() on all Append() calls
I20260621 05:10:38.921864 8173 tablet_bootstrap.cc:492] T 9c241e1876e04e84af26690f22910c79 P f47decfebb6c4a78a0f7a9126f201e08: No bootstrap required, opened a new log
I20260621 05:10:38.921937 8173 ts_tablet_manager.cc:1397] T 9c241e1876e04e84af26690f22910c79 P f47decfebb6c4a78a0f7a9126f201e08: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260621 05:10:38.923097 8175 tablet_bootstrap.cc:492] T 9c241e1876e04e84af26690f22910c79 P 4d3490def2e54588b8d71a7493761370: Bootstrap starting.
I20260621 05:10:38.923570 8176 tablet_bootstrap.cc:492] T 9c241e1876e04e84af26690f22910c79 P 7e1d10dd516045ad8959b889d7ac422e: Bootstrap starting.
I20260621 05:10:38.923915 8175 tablet_bootstrap.cc:654] T 9c241e1876e04e84af26690f22910c79 P 4d3490def2e54588b8d71a7493761370: Neither blocks nor log segments found. Creating new log.
I20260621 05:10:38.924235 8176 tablet_bootstrap.cc:654] T 9c241e1876e04e84af26690f22910c79 P 7e1d10dd516045ad8959b889d7ac422e: Neither blocks nor log segments found. Creating new log.
I20260621 05:10:38.924108 8173 raft_consensus.cc:348] T 9c241e1876e04e84af26690f22910c79 P f47decfebb6c4a78a0f7a9126f201e08 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d3490def2e54588b8d71a7493761370" member_type: VOTER last_known_addr { host: "127.3.131.129" port: 33077 } } peers { permanent_uuid: "7e1d10dd516045ad8959b889d7ac422e" member_type: VOTER last_known_addr { host: "127.3.131.131" port: 38381 } } peers { permanent_uuid: "f47decfebb6c4a78a0f7a9126f201e08" member_type: VOTER last_known_addr { host: "127.3.131.130" port: 43689 } }
I20260621 05:10:38.924557 8173 raft_consensus.cc:374] T 9c241e1876e04e84af26690f22910c79 P f47decfebb6c4a78a0f7a9126f201e08 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 05:10:38.924556 8175 log.cc:824] T 9c241e1876e04e84af26690f22910c79 P 4d3490def2e54588b8d71a7493761370: Log is configured to *not* fsync() on all Append() calls
I20260621 05:10:38.924610 8173 raft_consensus.cc:729] T 9c241e1876e04e84af26690f22910c79 P f47decfebb6c4a78a0f7a9126f201e08 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f47decfebb6c4a78a0f7a9126f201e08, State: Initialized, Role: FOLLOWER
I20260621 05:10:38.924731 8173 consensus_queue.cc:260] T 9c241e1876e04e84af26690f22910c79 P f47decfebb6c4a78a0f7a9126f201e08 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d3490def2e54588b8d71a7493761370" member_type: VOTER last_known_addr { host: "127.3.131.129" port: 33077 } } peers { permanent_uuid: "7e1d10dd516045ad8959b889d7ac422e" member_type: VOTER last_known_addr { host: "127.3.131.131" port: 38381 } } peers { permanent_uuid: "f47decfebb6c4a78a0f7a9126f201e08" member_type: VOTER last_known_addr { host: "127.3.131.130" port: 43689 } }
I20260621 05:10:38.924989 8176 log.cc:824] T 9c241e1876e04e84af26690f22910c79 P 7e1d10dd516045ad8959b889d7ac422e: Log is configured to *not* fsync() on all Append() calls
I20260621 05:10:38.925042 8173 ts_tablet_manager.cc:1428] T 9c241e1876e04e84af26690f22910c79 P f47decfebb6c4a78a0f7a9126f201e08: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260621 05:10:38.925163 8023 heartbeater.cc:503] Master 127.3.131.190:35757 was elected leader, sending a full tablet report...
I20260621 05:10:38.925932 8175 tablet_bootstrap.cc:492] T 9c241e1876e04e84af26690f22910c79 P 4d3490def2e54588b8d71a7493761370: No bootstrap required, opened a new log
I20260621 05:10:38.926002 8175 ts_tablet_manager.cc:1397] T 9c241e1876e04e84af26690f22910c79 P 4d3490def2e54588b8d71a7493761370: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260621 05:10:38.927696 8176 tablet_bootstrap.cc:492] T 9c241e1876e04e84af26690f22910c79 P 7e1d10dd516045ad8959b889d7ac422e: No bootstrap required, opened a new log
I20260621 05:10:38.927773 8176 ts_tablet_manager.cc:1397] T 9c241e1876e04e84af26690f22910c79 P 7e1d10dd516045ad8959b889d7ac422e: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260621 05:10:38.927753 8175 raft_consensus.cc:348] T 9c241e1876e04e84af26690f22910c79 P 4d3490def2e54588b8d71a7493761370 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d3490def2e54588b8d71a7493761370" member_type: VOTER last_known_addr { host: "127.3.131.129" port: 33077 } } peers { permanent_uuid: "7e1d10dd516045ad8959b889d7ac422e" member_type: VOTER last_known_addr { host: "127.3.131.131" port: 38381 } } peers { permanent_uuid: "f47decfebb6c4a78a0f7a9126f201e08" member_type: VOTER last_known_addr { host: "127.3.131.130" port: 43689 } }
I20260621 05:10:38.927917 8175 raft_consensus.cc:374] T 9c241e1876e04e84af26690f22910c79 P 4d3490def2e54588b8d71a7493761370 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 05:10:38.927951 8175 raft_consensus.cc:729] T 9c241e1876e04e84af26690f22910c79 P 4d3490def2e54588b8d71a7493761370 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d3490def2e54588b8d71a7493761370, State: Initialized, Role: FOLLOWER
I20260621 05:10:38.928081 8175 consensus_queue.cc:260] T 9c241e1876e04e84af26690f22910c79 P 4d3490def2e54588b8d71a7493761370 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d3490def2e54588b8d71a7493761370" member_type: VOTER last_known_addr { host: "127.3.131.129" port: 33077 } } peers { permanent_uuid: "7e1d10dd516045ad8959b889d7ac422e" member_type: VOTER last_known_addr { host: "127.3.131.131" port: 38381 } } peers { permanent_uuid: "f47decfebb6c4a78a0f7a9126f201e08" member_type: VOTER last_known_addr { host: "127.3.131.130" port: 43689 } }
I20260621 05:10:38.928371 8175 ts_tablet_manager.cc:1428] T 9c241e1876e04e84af26690f22910c79 P 4d3490def2e54588b8d71a7493761370: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260621 05:10:38.928418 7890 heartbeater.cc:503] Master 127.3.131.190:35757 was elected leader, sending a full tablet report...
I20260621 05:10:38.929553 8176 raft_consensus.cc:348] T 9c241e1876e04e84af26690f22910c79 P 7e1d10dd516045ad8959b889d7ac422e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d3490def2e54588b8d71a7493761370" member_type: VOTER last_known_addr { host: "127.3.131.129" port: 33077 } } peers { permanent_uuid: "7e1d10dd516045ad8959b889d7ac422e" member_type: VOTER last_known_addr { host: "127.3.131.131" port: 38381 } } peers { permanent_uuid: "f47decfebb6c4a78a0f7a9126f201e08" member_type: VOTER last_known_addr { host: "127.3.131.130" port: 43689 } }
I20260621 05:10:38.929682 8176 raft_consensus.cc:374] T 9c241e1876e04e84af26690f22910c79 P 7e1d10dd516045ad8959b889d7ac422e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 05:10:38.929708 8176 raft_consensus.cc:729] T 9c241e1876e04e84af26690f22910c79 P 7e1d10dd516045ad8959b889d7ac422e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7e1d10dd516045ad8959b889d7ac422e, State: Initialized, Role: FOLLOWER
I20260621 05:10:38.929826 8176 consensus_queue.cc:260] T 9c241e1876e04e84af26690f22910c79 P 7e1d10dd516045ad8959b889d7ac422e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d3490def2e54588b8d71a7493761370" member_type: VOTER last_known_addr { host: "127.3.131.129" port: 33077 } } peers { permanent_uuid: "7e1d10dd516045ad8959b889d7ac422e" member_type: VOTER last_known_addr { host: "127.3.131.131" port: 38381 } } peers { permanent_uuid: "f47decfebb6c4a78a0f7a9126f201e08" member_type: VOTER last_known_addr { host: "127.3.131.130" port: 43689 } }
I20260621 05:10:38.930104 8176 ts_tablet_manager.cc:1428] T 9c241e1876e04e84af26690f22910c79 P 7e1d10dd516045ad8959b889d7ac422e: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260621 05:10:38.930116 8156 heartbeater.cc:503] Master 127.3.131.190:35757 was elected leader, sending a full tablet report...
W20260621 05:10:38.960536 8024 tablet.cc:2369] T 9c241e1876e04e84af26690f22910c79 P f47decfebb6c4a78a0f7a9126f201e08: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260621 05:10:39.002389 8177 raft_consensus.cc:482] T 9c241e1876e04e84af26690f22910c79 P f47decfebb6c4a78a0f7a9126f201e08 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260621 05:10:39.002522 8177 raft_consensus.cc:504] T 9c241e1876e04e84af26690f22910c79 P f47decfebb6c4a78a0f7a9126f201e08 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d3490def2e54588b8d71a7493761370" member_type: VOTER last_known_addr { host: "127.3.131.129" port: 33077 } } peers { permanent_uuid: "7e1d10dd516045ad8959b889d7ac422e" member_type: VOTER last_known_addr { host: "127.3.131.131" port: 38381 } } peers { permanent_uuid: "f47decfebb6c4a78a0f7a9126f201e08" member_type: VOTER last_known_addr { host: "127.3.131.130" port: 43689 } }
I20260621 05:10:39.002935 8177 leader_election.cc:288] T 9c241e1876e04e84af26690f22910c79 P f47decfebb6c4a78a0f7a9126f201e08 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4d3490def2e54588b8d71a7493761370 (127.3.131.129:33077), 7e1d10dd516045ad8959b889d7ac422e (127.3.131.131:38381)
I20260621 05:10:39.005831 8170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:10:39.003263 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.131.130:60589 (local address 127.3.131.129:33077)
0621 05:10:39.003325 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0621 05:10:39.003327 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:10:39.003338 (+ 11us) server_negotiation.cc:407] Connection header received
0621 05:10:39.003371 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:10:39.003372 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:10:39.003381 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:10:39.003474 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 05:10:39.003666 (+ 192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:10:39.004145 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:10:39.005117 (+ 972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:10:39.005472 (+ 355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:10:39.005554 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:10:39.005564 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0621 05:10:39.005728 (+ 164us) server_negotiation.cc:299] Negotiation successful
0621 05:10:39.005764 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260621 05:10:39.005831 8025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:10:39.003129 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.131.129:33077 (local address 127.3.131.130:60589)
0621 05:10:39.003228 (+ 99us) negotiation.cc:107] Waiting for socket to connect
0621 05:10:39.003242 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0621 05:10:39.003288 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:10:39.003478 (+ 190us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:10:39.003480 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:10:39.003482 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 05:10:39.003559 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:10:39.003560 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:10:39.004230 (+ 670us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:10:39.004231 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:10:39.005003 (+ 772us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:10:39.005005 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:10:39.005549 (+ 544us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:10:39.005550 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:10:39.005643 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:10:39.005646 (+ 3us) client_negotiation.cc:769] Sending connection context
0621 05:10:39.005709 (+ 63us) client_negotiation.cc:240] Negotiation successful
0621 05:10:39.005756 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":75}
I20260621 05:10:39.006075 8172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:10:39.003428 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.131.130:51797 (local address 127.3.131.131:38381)
0621 05:10:39.003476 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0621 05:10:39.003478 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:10:39.003542 (+ 64us) server_negotiation.cc:407] Connection header received
0621 05:10:39.003619 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:10:39.003620 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:10:39.003629 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:10:39.003698 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 05:10:39.003846 (+ 148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:10:39.004342 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:10:39.005285 (+ 943us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:10:39.005669 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:10:39.005753 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:10:39.005761 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0621 05:10:39.005947 (+ 186us) server_negotiation.cc:299] Negotiation successful
0621 05:10:39.006002 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260621 05:10:39.006182 7844 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c241e1876e04e84af26690f22910c79" candidate_uuid: "f47decfebb6c4a78a0f7a9126f201e08" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d3490def2e54588b8d71a7493761370" is_pre_election: true
I20260621 05:10:39.006294 8182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:10:39.003306 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.131.131:38381 (local address 127.3.131.130:51797)
0621 05:10:39.003490 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0621 05:10:39.003499 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0621 05:10:39.003556 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:10:39.003700 (+ 144us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:10:39.003701 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:10:39.003702 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 05:10:39.003776 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:10:39.003778 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:10:39.004442 (+ 664us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:10:39.004443 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:10:39.005187 (+ 744us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:10:39.005189 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:10:39.005756 (+ 567us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:10:39.005757 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:10:39.005834 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:10:39.005837 (+ 3us) client_negotiation.cc:769] Sending connection context
0621 05:10:39.006171 (+ 334us) client_negotiation.cc:240] Negotiation successful
0621 05:10:39.006217 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":170,"thread_start_us":121,"threads_started":1}
I20260621 05:10:39.006364 7844 raft_consensus.cc:2454] T 9c241e1876e04e84af26690f22910c79 P 4d3490def2e54588b8d71a7493761370 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f47decfebb6c4a78a0f7a9126f201e08 in term 0.
I20260621 05:10:39.006613 7911 leader_election.cc:302] T 9c241e1876e04e84af26690f22910c79 P f47decfebb6c4a78a0f7a9126f201e08 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d3490def2e54588b8d71a7493761370, f47decfebb6c4a78a0f7a9126f201e08; no voters:
I20260621 05:10:39.006758 8110 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c241e1876e04e84af26690f22910c79" candidate_uuid: "f47decfebb6c4a78a0f7a9126f201e08" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7e1d10dd516045ad8959b889d7ac422e" is_pre_election: true
I20260621 05:10:39.006781 8177 raft_consensus.cc:2781] T 9c241e1876e04e84af26690f22910c79 P f47decfebb6c4a78a0f7a9126f201e08 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260621 05:10:39.006855 8177 raft_consensus.cc:482] T 9c241e1876e04e84af26690f22910c79 P f47decfebb6c4a78a0f7a9126f201e08 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260621 05:10:39.006888 8177 raft_consensus.cc:3037] T 9c241e1876e04e84af26690f22910c79 P f47decfebb6c4a78a0f7a9126f201e08 [term 0 FOLLOWER]: Advancing to term 1
I20260621 05:10:39.006935 8110 raft_consensus.cc:2454] T 9c241e1876e04e84af26690f22910c79 P 7e1d10dd516045ad8959b889d7ac422e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f47decfebb6c4a78a0f7a9126f201e08 in term 0.
I20260621 05:10:39.007779 8177 raft_consensus.cc:504] T 9c241e1876e04e84af26690f22910c79 P f47decfebb6c4a78a0f7a9126f201e08 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d3490def2e54588b8d71a7493761370" member_type: VOTER last_known_addr { host: "127.3.131.129" port: 33077 } } peers { permanent_uuid: "7e1d10dd516045ad8959b889d7ac422e" member_type: VOTER last_known_addr { host: "127.3.131.131" port: 38381 } } peers { permanent_uuid: "f47decfebb6c4a78a0f7a9126f201e08" member_type: VOTER last_known_addr { host: "127.3.131.130" port: 43689 } }
I20260621 05:10:39.007926 8177 leader_election.cc:288] T 9c241e1876e04e84af26690f22910c79 P f47decfebb6c4a78a0f7a9126f201e08 [CANDIDATE]: Term 1 election: Requested vote from peers 4d3490def2e54588b8d71a7493761370 (127.3.131.129:33077), 7e1d10dd516045ad8959b889d7ac422e (127.3.131.131:38381)
I20260621 05:10:39.008133 7844 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c241e1876e04e84af26690f22910c79" candidate_uuid: "f47decfebb6c4a78a0f7a9126f201e08" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d3490def2e54588b8d71a7493761370"
I20260621 05:10:39.008224 8110 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c241e1876e04e84af26690f22910c79" candidate_uuid: "f47decfebb6c4a78a0f7a9126f201e08" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7e1d10dd516045ad8959b889d7ac422e"
I20260621 05:10:39.008239 7844 raft_consensus.cc:3037] T 9c241e1876e04e84af26690f22910c79 P 4d3490def2e54588b8d71a7493761370 [term 0 FOLLOWER]: Advancing to term 1
I20260621 05:10:39.008316 8110 raft_consensus.cc:3037] T 9c241e1876e04e84af26690f22910c79 P 7e1d10dd516045ad8959b889d7ac422e [term 0 FOLLOWER]: Advancing to term 1
I20260621 05:10:39.008951 7844 raft_consensus.cc:2454] T 9c241e1876e04e84af26690f22910c79 P 4d3490def2e54588b8d71a7493761370 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f47decfebb6c4a78a0f7a9126f201e08 in term 1.
I20260621 05:10:39.009158 7911 leader_election.cc:302] T 9c241e1876e04e84af26690f22910c79 P f47decfebb6c4a78a0f7a9126f201e08 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d3490def2e54588b8d71a7493761370, f47decfebb6c4a78a0f7a9126f201e08; no voters:
I20260621 05:10:39.009284 8177 raft_consensus.cc:2781] T 9c241e1876e04e84af26690f22910c79 P f47decfebb6c4a78a0f7a9126f201e08 [term 1 FOLLOWER]: Leader election won for term 1
I20260621 05:10:39.009303 8110 raft_consensus.cc:2454] T 9c241e1876e04e84af26690f22910c79 P 7e1d10dd516045ad8959b889d7ac422e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f47decfebb6c4a78a0f7a9126f201e08 in term 1.
I20260621 05:10:39.009478 8177 raft_consensus.cc:686] T 9c241e1876e04e84af26690f22910c79 P f47decfebb6c4a78a0f7a9126f201e08 [term 1 LEADER]: Becoming Leader. State: Replica: f47decfebb6c4a78a0f7a9126f201e08, State: Running, Role: LEADER
I20260621 05:10:39.009581 8177 consensus_queue.cc:237] T 9c241e1876e04e84af26690f22910c79 P f47decfebb6c4a78a0f7a9126f201e08 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d3490def2e54588b8d71a7493761370" member_type: VOTER last_known_addr { host: "127.3.131.129" port: 33077 } } peers { permanent_uuid: "7e1d10dd516045ad8959b889d7ac422e" member_type: VOTER last_known_addr { host: "127.3.131.131" port: 38381 } } peers { permanent_uuid: "f47decfebb6c4a78a0f7a9126f201e08" member_type: VOTER last_known_addr { host: "127.3.131.130" port: 43689 } }
I20260621 05:10:39.010392 7699 catalog_manager.cc:5495] T 9c241e1876e04e84af26690f22910c79 P f47decfebb6c4a78a0f7a9126f201e08 reported cstate change: term changed from 0 to 1, leader changed from <none> to f47decfebb6c4a78a0f7a9126f201e08 (127.3.131.130). New cstate: current_term: 1 leader_uuid: "f47decfebb6c4a78a0f7a9126f201e08" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d3490def2e54588b8d71a7493761370" member_type: VOTER last_known_addr { host: "127.3.131.129" port: 33077 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7e1d10dd516045ad8959b889d7ac422e" member_type: VOTER last_known_addr { host: "127.3.131.131" port: 38381 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f47decfebb6c4a78a0f7a9126f201e08" member_type: VOTER last_known_addr { host: "127.3.131.130" port: 43689 } health_report { overall_health: HEALTHY } } }
I20260621 05:10:39.035960 8169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:10:39.033164 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59768 (local address 127.3.131.130:43689)
0621 05:10:39.033224 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0621 05:10:39.033227 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0621 05:10:39.033259 (+ 32us) server_negotiation.cc:407] Connection header received
0621 05:10:39.033303 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:10:39.033304 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:10:39.033322 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:10:39.033375 (+ 53us) server_negotiation.cc:226] Negotiated authn=TOKEN
0621 05:10:39.033566 (+ 191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:10:39.034080 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:10:39.034890 (+ 810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:10:39.035114 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:10:39.035282 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:10:39.035455 (+ 173us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0621 05:10:39.035589 (+ 134us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0621 05:10:39.035677 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0621 05:10:39.035781 (+ 104us) server_negotiation.cc:299] Negotiation successful
0621 05:10:39.035804 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260621 05:10:39.038427 8110 raft_consensus.cc:1261] T 9c241e1876e04e84af26690f22910c79 P 7e1d10dd516045ad8959b889d7ac422e [term 1 FOLLOWER]: Refusing update from remote peer f47decfebb6c4a78a0f7a9126f201e08: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 05:10:39.038442 7844 raft_consensus.cc:1261] T 9c241e1876e04e84af26690f22910c79 P 4d3490def2e54588b8d71a7493761370 [term 1 FOLLOWER]: Refusing update from remote peer f47decfebb6c4a78a0f7a9126f201e08: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 05:10:39.038830 8177 consensus_queue.cc:1035] T 9c241e1876e04e84af26690f22910c79 P f47decfebb6c4a78a0f7a9126f201e08 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7e1d10dd516045ad8959b889d7ac422e" member_type: VOTER last_known_addr { host: "127.3.131.131" port: 38381 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 05:10:39.039026 8183 consensus_queue.cc:1035] T 9c241e1876e04e84af26690f22910c79 P f47decfebb6c4a78a0f7a9126f201e08 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d3490def2e54588b8d71a7493761370" member_type: VOTER last_known_addr { host: "127.3.131.129" port: 33077 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260621 05:10:39.040813 7891 tablet.cc:2369] T 9c241e1876e04e84af26690f22910c79 P 4d3490def2e54588b8d71a7493761370: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260621 05:10:39.042719 8187 mvcc.cc:205] Tried to move back new op lower bound from 7299148345496879104 to 7299148345383804928. Current Snapshot: MvccSnapshot[applied={T|T < 7299148345496879104}]
I20260621 05:10:39.042990 8188 mvcc.cc:205] Tried to move back new op lower bound from 7299148345496879104 to 7299148345383804928. Current Snapshot: MvccSnapshot[applied={T|T < 7299148345496879104}]
I20260621 05:10:39.045796 7700 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47796:
table { table_name: "test-table" } modify_external_catalogs: true
I20260621 05:10:39.045943 7700 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47796:
table { table_name: "test-table" } modify_external_catalogs: true
I20260621 05:10:39.046810 7700 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2f959bd270b8497a85a36fbdfea7ca32: Sending DeleteTablet for 3 replicas of tablet 9c241e1876e04e84af26690f22910c79
I20260621 05:10:39.047216 7824 tablet_service.cc:1514] Processing DeleteTablet for tablet 9c241e1876e04e84af26690f22910c79 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 05:10:39 UTC) from {username='kudu', principal='kudu/127.3.131.190@KRBTEST.COM'} at 127.0.0.1:49208
I20260621 05:10:39.047255 7957 tablet_service.cc:1514] Processing DeleteTablet for tablet 9c241e1876e04e84af26690f22910c79 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 05:10:39 UTC) from {username='kudu', principal='kudu/127.3.131.190@KRBTEST.COM'} at 127.0.0.1:59756
I20260621 05:10:39.047272 8090 tablet_service.cc:1514] Processing DeleteTablet for tablet 9c241e1876e04e84af26690f22910c79 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 05:10:39 UTC) from {username='kudu', principal='kudu/127.3.131.190@KRBTEST.COM'} at 127.0.0.1:42058
I20260621 05:10:39.047528 8195 tablet_replica.cc:330] T 9c241e1876e04e84af26690f22910c79 P 4d3490def2e54588b8d71a7493761370: stopping tablet replica
I20260621 05:10:39.047653 8195 raft_consensus.cc:2229] T 9c241e1876e04e84af26690f22910c79 P 4d3490def2e54588b8d71a7493761370 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260621 05:10:39.047756 3598 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWZOx2_/build/release/bin/kudu with pid 7762
I20260621 05:10:39.047798 8195 raft_consensus.cc:2258] T 9c241e1876e04e84af26690f22910c79 P 4d3490def2e54588b8d71a7493761370 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 05:10:39.048355 8197 tablet_replica.cc:330] T 9c241e1876e04e84af26690f22910c79 P f47decfebb6c4a78a0f7a9126f201e08: stopping tablet replica
I20260621 05:10:39.048451 8197 raft_consensus.cc:2229] T 9c241e1876e04e84af26690f22910c79 P f47decfebb6c4a78a0f7a9126f201e08 [term 1 LEADER]: Raft consensus shutting down.
I20260621 05:10:39.048584 8197 raft_consensus.cc:2258] T 9c241e1876e04e84af26690f22910c79 P f47decfebb6c4a78a0f7a9126f201e08 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 05:10:39.048775 8196 tablet_replica.cc:330] T 9c241e1876e04e84af26690f22910c79 P 7e1d10dd516045ad8959b889d7ac422e: stopping tablet replica
I20260621 05:10:39.048907 8196 raft_consensus.cc:2229] T 9c241e1876e04e84af26690f22910c79 P 7e1d10dd516045ad8959b889d7ac422e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260621 05:10:39.049018 8196 raft_consensus.cc:2258] T 9c241e1876e04e84af26690f22910c79 P 7e1d10dd516045ad8959b889d7ac422e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 05:10:39.049623 8196 ts_tablet_manager.cc:1905] T 9c241e1876e04e84af26690f22910c79 P 7e1d10dd516045ad8959b889d7ac422e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 05:10:39.049950 8197 ts_tablet_manager.cc:1905] T 9c241e1876e04e84af26690f22910c79 P f47decfebb6c4a78a0f7a9126f201e08: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 05:10:39.051890 8196 ts_tablet_manager.cc:1918] T 9c241e1876e04e84af26690f22910c79 P 7e1d10dd516045ad8959b889d7ac422e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260621 05:10:39.051901 8197 ts_tablet_manager.cc:1918] T 9c241e1876e04e84af26690f22910c79 P f47decfebb6c4a78a0f7a9126f201e08: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260621 05:10:39.051975 8197 log.cc:1196] T 9c241e1876e04e84af26690f22910c79 P f47decfebb6c4a78a0f7a9126f201e08: Deleting WAL directory at /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782018616648578-3598-0/minicluster-data/ts-1/wal/wals/9c241e1876e04e84af26690f22910c79
I20260621 05:10:39.052187 8196 log.cc:1196] T 9c241e1876e04e84af26690f22910c79 P 7e1d10dd516045ad8959b889d7ac422e: Deleting WAL directory at /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782018616648578-3598-0/minicluster-data/ts-2/wal/wals/9c241e1876e04e84af26690f22910c79
I20260621 05:10:39.052258 8197 ts_tablet_manager.cc:1939] T 9c241e1876e04e84af26690f22910c79 P f47decfebb6c4a78a0f7a9126f201e08: Deleting consensus metadata
I20260621 05:10:39.052588 8196 ts_tablet_manager.cc:1939] T 9c241e1876e04e84af26690f22910c79 P 7e1d10dd516045ad8959b889d7ac422e: Deleting consensus metadata
I20260621 05:10:39.052800 7687 catalog_manager.cc:4841] TS f47decfebb6c4a78a0f7a9126f201e08 (127.3.131.130:43689): tablet 9c241e1876e04e84af26690f22910c79 (table test-table [id=f6291787ef82483991f83f2c184d0b48]) successfully deleted
I20260621 05:10:39.053295 7685 catalog_manager.cc:4841] TS 7e1d10dd516045ad8959b889d7ac422e (127.3.131.131:38381): tablet 9c241e1876e04e84af26690f22910c79 (table test-table [id=f6291787ef82483991f83f2c184d0b48]) successfully deleted
W20260621 05:10:39.056334 7685 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.3.131.129:33077 (error 108)
I20260621 05:10:39.056555 3598 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWZOx2_/build/release/bin/kudu with pid 7895
I20260621 05:10:39.057762 8167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:10:39.057591 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.131.129:33077 (local address 127.0.0.1:49214)
0621 05:10:39.057668 (+ 77us) negotiation.cc:107] Waiting for socket to connect
0621 05:10:39.057717 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.131.129:33077: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260621 05:10:39.058007 7685 catalog_manager.cc:4584] TS 4d3490def2e54588b8d71a7493761370 (127.3.131.129:33077): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 9c241e1876e04e84af26690f22910c79: Network error: Client connection negotiation failed: client connection to 127.3.131.129:33077: connect: Connection refused (error 111)
I20260621 05:10:39.061368 3598 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWZOx2_/build/release/bin/kudu with pid 8028
I20260621 05:10:39.065745 3598 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWZOx2_/build/release/bin/kudu with pid 7669
2026-06-21T05:10:39Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (2074 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-06-21T05:10:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-21T05:10:39Z Disabled control of system clock
I20260621 05:10:39.094898 3598 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWZOx2_/build/release/bin/kudu
/tmp/dist-test-taskWZOx2_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782018616648578-3598-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782018616648578-3598-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782018616648578-3598-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782018616648578-3598-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.131.190:36105
--webserver_interface=127.3.131.190
--webserver_port=0
--builtin_ntp_servers=127.3.131.148:44959
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.131.190:36105
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260621 05:10:39.178655 8202 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 05:10:39.179035 8202 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 05:10:39.179065 8202 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 05:10:39.180552 8202 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 05:10:39.180609 8202 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 05:10:39.180622 8202 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 05:10:39.180636 8202 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 05:10:39.180648 8202 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 05:10:39.182190 8202 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.131.148:44959
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782018616648578-3598-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782018616648578-3598-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.131.190:36105
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.131.190:36105
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782018616648578-3598-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.131.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8202
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782018616648578-3598-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 21 Jun 2026 04:16:16 UTC on ccycloud.quasar-xycaha.root.comops.site
build id 4457881
I20260621 05:10:39.182518 8202 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 05:10:39.182916 8202 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 05:10:39.188117 8210 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:10:39.188189 8207 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:10:39.188117 8208 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:10:39.188656 8202 server_base.cc:1034] running on GCE node
I20260621 05:10:39.188933 8202 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 05:10:39.189312 8202 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 05:10:39.190498 8202 hybrid_clock.cc:648] HybridClock initialized: now 1782018639190486 us; error 30 us; skew 500 ppm
I20260621 05:10:39.191953 8202 webserver.cc:460] Webserver started at http://127.3.131.190:41283/ using document root <none> and password file <none>
I20260621 05:10:39.192278 8202 fs_manager.cc:359] Metadata directory not provided
I20260621 05:10:39.192354 8202 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 05:10:39.192507 8202 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 05:10:39.193616 8202 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782018616648578-3598-0/minicluster-data/master-0/data/instance:
uuid: "7f7b32217bc84f5c938dbd1da1cc85a4"
format_stamp: "Formatted at 2026-06-21 05:10:39 on dist-test-slave-vrq1"
server_key: "abdc7458797ac63cf7837dbc3c1da0b5"
server_key_iv: "2a31d46c2b29e2b88f54272bb80758dd"
server_key_version: "encryptionkey@0"
I20260621 05:10:39.193984 8202 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782018616648578-3598-0/minicluster-data/master-0/wal/instance:
uuid: "7f7b32217bc84f5c938dbd1da1cc85a4"
format_stamp: "Formatted at 2026-06-21 05:10:39 on dist-test-slave-vrq1"
server_key: "abdc7458797ac63cf7837dbc3c1da0b5"
server_key_iv: "2a31d46c2b29e2b88f54272bb80758dd"
server_key_version: "encryptionkey@0"
I20260621 05:10:39.195565 8202 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260621 05:10:39.196523 8216 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 05:10:39.196718 8202 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260621 05:10:39.196822 8202 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782018616648578-3598-0/minicluster-data/master-0/data,/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782018616648578-3598-0/minicluster-data/master-0/wal
uuid: "7f7b32217bc84f5c938dbd1da1cc85a4"
format_stamp: "Formatted at 2026-06-21 05:10:39 on dist-test-slave-vrq1"
server_key: "abdc7458797ac63cf7837dbc3c1da0b5"
server_key_iv: "2a31d46c2b29e2b88f54272bb80758dd"
server_key_version: "encryptionkey@0"
I20260621 05:10:39.196908 8202 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782018616648578-3598-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782018616648578-3598-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782018616648578-3598-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 05:10:39.200726 8202 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 05:10:39.201026 8202 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 05:10:39.201179 8202 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 05:10:39.205485 8202 rpc_server.cc:307] RPC server started. Bound to: 127.3.131.190:36105
I20260621 05:10:39.205610 8268 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.131.190:36105 every 8 connection(s)
I20260621 05:10:39.205932 8202 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782018616648578-3598-0/minicluster-data/master-0/data/info.pb
I20260621 05:10:39.206897 8269 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 05:10:39.209868 3598 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWZOx2_/build/release/bin/kudu as pid 8202
I20260621 05:10:39.210013 3598 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782018616648578-3598-0/minicluster-data/master-0/wal/instance
I20260621 05:10:39.210181 3598 external_mini_cluster.cc:1428] Setting key 81f65e725350ec16dda9579616378a9f
I20260621 05:10:39.210186 8269 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7f7b32217bc84f5c938dbd1da1cc85a4: Bootstrap starting.
I20260621 05:10:39.210994 8269 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7f7b32217bc84f5c938dbd1da1cc85a4: Neither blocks nor log segments found. Creating new log.
I20260621 05:10:39.211361 8269 log.cc:824] T 00000000000000000000000000000000 P 7f7b32217bc84f5c938dbd1da1cc85a4: Log is configured to *not* fsync() on all Append() calls
I20260621 05:10:39.212216 8269 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7f7b32217bc84f5c938dbd1da1cc85a4: No bootstrap required, opened a new log
I20260621 05:10:39.213339 8272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:10:39.210823 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50662 (local address 127.3.131.190:36105)
0621 05:10:39.211052 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0621 05:10:39.211061 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0621 05:10:39.211098 (+ 37us) server_negotiation.cc:407] Connection header received
0621 05:10:39.211757 (+ 659us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:10:39.211765 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:10:39.211807 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:10:39.211990 (+ 183us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:10:39.212741 (+ 751us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:10:39.212745 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:10:39.212746 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 05:10:39.212750 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:10:39.212801 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 05:10:39.212822 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:10:39.212919 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0621 05:10:39.213064 (+ 145us) server_negotiation.cc:299] Negotiation successful
0621 05:10:39.213137 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":143,"thread_start_us":88,"threads_started":1}
I20260621 05:10:39.214074 8269 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7f7b32217bc84f5c938dbd1da1cc85a4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f7b32217bc84f5c938dbd1da1cc85a4" member_type: VOTER last_known_addr { host: "127.3.131.190" port: 36105 } }
I20260621 05:10:39.214246 8269 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7f7b32217bc84f5c938dbd1da1cc85a4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 05:10:39.214293 8269 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7f7b32217bc84f5c938dbd1da1cc85a4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7f7b32217bc84f5c938dbd1da1cc85a4, State: Initialized, Role: FOLLOWER
I20260621 05:10:39.214428 8269 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7f7b32217bc84f5c938dbd1da1cc85a4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f7b32217bc84f5c938dbd1da1cc85a4" member_type: VOTER last_known_addr { host: "127.3.131.190" port: 36105 } }
I20260621 05:10:39.214545 8269 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7f7b32217bc84f5c938dbd1da1cc85a4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260621 05:10:39.214587 8269 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7f7b32217bc84f5c938dbd1da1cc85a4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260621 05:10:39.214645 8269 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7f7b32217bc84f5c938dbd1da1cc85a4 [term 0 FOLLOWER]: Advancing to term 1
I20260621 05:10:39.215628 8269 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7f7b32217bc84f5c938dbd1da1cc85a4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f7b32217bc84f5c938dbd1da1cc85a4" member_type: VOTER last_known_addr { host: "127.3.131.190" port: 36105 } }
I20260621 05:10:39.215770 8269 leader_election.cc:302] T 00000000000000000000000000000000 P 7f7b32217bc84f5c938dbd1da1cc85a4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7f7b32217bc84f5c938dbd1da1cc85a4; no voters:
I20260621 05:10:39.215956 8269 leader_election.cc:288] T 00000000000000000000000000000000 P 7f7b32217bc84f5c938dbd1da1cc85a4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260621 05:10:39.216005 8274 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7f7b32217bc84f5c938dbd1da1cc85a4 [term 1 FOLLOWER]: Leader election won for term 1
I20260621 05:10:39.216235 8269 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7f7b32217bc84f5c938dbd1da1cc85a4 [sys.catalog]: configured and running, proceeding with master startup.
I20260621 05:10:39.216287 8274 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7f7b32217bc84f5c938dbd1da1cc85a4 [term 1 LEADER]: Becoming Leader. State: Replica: 7f7b32217bc84f5c938dbd1da1cc85a4, State: Running, Role: LEADER
I20260621 05:10:39.216405 8274 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7f7b32217bc84f5c938dbd1da1cc85a4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f7b32217bc84f5c938dbd1da1cc85a4" member_type: VOTER last_known_addr { host: "127.3.131.190" port: 36105 } }
I20260621 05:10:39.216823 8275 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7f7b32217bc84f5c938dbd1da1cc85a4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7f7b32217bc84f5c938dbd1da1cc85a4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f7b32217bc84f5c938dbd1da1cc85a4" member_type: VOTER last_known_addr { host: "127.3.131.190" port: 36105 } } }
I20260621 05:10:39.216938 8275 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7f7b32217bc84f5c938dbd1da1cc85a4 [sys.catalog]: This master's current role is: LEADER
I20260621 05:10:39.217124 8276 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7f7b32217bc84f5c938dbd1da1cc85a4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7f7b32217bc84f5c938dbd1da1cc85a4. Latest consensus state: current_term: 1 leader_uuid: "7f7b32217bc84f5c938dbd1da1cc85a4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f7b32217bc84f5c938dbd1da1cc85a4" member_type: VOTER last_known_addr { host: "127.3.131.190" port: 36105 } } }
I20260621 05:10:39.217190 8276 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7f7b32217bc84f5c938dbd1da1cc85a4 [sys.catalog]: This master's current role is: LEADER
I20260621 05:10:39.217206 8283 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260621 05:10:39.217749 8283 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260621 05:10:39.219398 8283 catalog_manager.cc:1347] Generated new cluster ID: 23ad06b47f774408a4b753497b9b9f71
I20260621 05:10:39.219450 8283 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260621 05:10:39.235005 8283 catalog_manager.cc:1370] Generated new certificate authority record
I20260621 05:10:39.235250 8283 catalog_manager.cc:1504] Loading token signing keys...
I20260621 05:10:39.240893 8283 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7f7b32217bc84f5c938dbd1da1cc85a4: Generated new TSK 0
I20260621 05:10:39.241117 8283 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260621 05:10:39.246966 3598 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWZOx2_/build/release/bin/kudu
/tmp/dist-test-taskWZOx2_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782018616648578-3598-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782018616648578-3598-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782018616648578-3598-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782018616648578-3598-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.131.129:0
--local_ip_for_outbound_sockets=127.3.131.129
--webserver_interface=127.3.131.129
--webserver_port=0
--tserver_master_addrs=127.3.131.190:36105
--builtin_ntp_servers=127.3.131.148:44959
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260621 05:10:39.340829 8293 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 05:10:39.341074 8293 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 05:10:39.341099 8293 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 05:10:39.342403 8293 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 05:10:39.342444 8293 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 05:10:39.342471 8293 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.131.129
I20260621 05:10:39.343961 8293 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.131.148:44959
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782018616648578-3598-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782018616648578-3598-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.131.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782018616648578-3598-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.131.129
--webserver_port=0
--tserver_master_addrs=127.3.131.190:36105
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8293
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.131.129
--log_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782018616648578-3598-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 21 Jun 2026 04:16:16 UTC on ccycloud.quasar-xycaha.root.comops.site
build id 4457881
I20260621 05:10:39.344221 8293 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 05:10:39.344548 8293 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 05:10:39.350572 8299 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:10:39.350559 8298 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:10:39.350553 8301 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:10:39.350819 8293 server_base.cc:1034] running on GCE node
I20260621 05:10:39.351140 8293 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 05:10:39.351459 8293 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 05:10:39.352641 8293 hybrid_clock.cc:648] HybridClock initialized: now 1782018639352615 us; error 39 us; skew 500 ppm
I20260621 05:10:39.354305 8293 webserver.cc:460] Webserver started at http://127.3.131.129:43929/ using document root <none> and password file <none>
I20260621 05:10:39.354578 8293 fs_manager.cc:359] Metadata directory not provided
I20260621 05:10:39.354658 8293 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 05:10:39.354831 8293 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 05:10:39.356083 8293 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782018616648578-3598-0/minicluster-data/ts-0/data/instance:
uuid: "3e5d2bacdfde4aba95becffd65fb03fb"
format_stamp: "Formatted at 2026-06-21 05:10:39 on dist-test-slave-vrq1"
server_key: "00048c0c813b122df49882702c971489"
server_key_iv: "0238e644f875de23be1e754beddc5cec"
server_key_version: "encryptionkey@0"
I20260621 05:10:39.356484 8293 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782018616648578-3598-0/minicluster-data/ts-0/wal/instance:
uuid: "3e5d2bacdfde4aba95becffd65fb03fb"
format_stamp: "Formatted at 2026-06-21 05:10:39 on dist-test-slave-vrq1"
server_key: "00048c0c813b122df49882702c971489"
server_key_iv: "0238e644f875de23be1e754beddc5cec"
server_key_version: "encryptionkey@0"
I20260621 05:10:39.357975 8293 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260621 05:10:39.358840 8307 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 05:10:39.359025 8293 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260621 05:10:39.359134 8293 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782018616648578-3598-0/minicluster-data/ts-0/data,/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782018616648578-3598-0/minicluster-data/ts-0/wal
uuid: "3e5d2bacdfde4aba95becffd65fb03fb"
format_stamp: "Formatted at 2026-06-21 05:10:39 on dist-test-slave-vrq1"
server_key: "00048c0c813b122df49882702c971489"
server_key_iv: "0238e644f875de23be1e754beddc5cec"
server_key_version: "encryptionkey@0"
I20260621 05:10:39.359295 8293 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782018616648578-3598-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782018616648578-3598-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782018616648578-3598-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 05:10:39.362673 8293 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 05:10:39.362953 8293 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 05:10:39.363102 8293 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 05:10:39.363344 8293 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 05:10:39.363688 8293 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 05:10:39.363745 8293 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 05:10:39.363785 8293 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 05:10:39.363811 8293 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 05:10:39.369639 8293 rpc_server.cc:307] RPC server started. Bound to: 127.3.131.129:43157
I20260621 05:10:39.369738 8420 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.131.129:43157 every 8 connection(s)
I20260621 05:10:39.370078 8293 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782018616648578-3598-0/minicluster-data/ts-0/data/info.pb
I20260621 05:10:39.372792 3598 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWZOx2_/build/release/bin/kudu as pid 8293
I20260621 05:10:39.372879 3598 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782018616648578-3598-0/minicluster-data/ts-0/wal/instance
I20260621 05:10:39.373096 3598 external_mini_cluster.cc:1428] Setting key 2a2ea626ab113807deb2a85a06bd3ea3
I20260621 05:10:39.373806 8272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:10:39.370950 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.131.129:38779 (local address 127.3.131.190:36105)
0621 05:10:39.371019 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0621 05:10:39.371021 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:10:39.371369 (+ 348us) server_negotiation.cc:407] Connection header received
0621 05:10:39.371864 (+ 495us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:10:39.371865 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:10:39.371871 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:10:39.371939 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:10:39.373304 (+ 1365us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:10:39.373306 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:10:39.373306 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 05:10:39.373311 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:10:39.373334 (+ 23us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 05:10:39.373339 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:10:39.373422 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0621 05:10:39.373734 (+ 312us) server_negotiation.cc:299] Negotiation successful
0621 05:10:39.373749 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260621 05:10:39.374279 8423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:10:39.370947 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.131.190:36105 (local address 127.3.131.129:38779)
0621 05:10:39.371135 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0621 05:10:39.371269 (+ 134us) client_negotiation.cc:174] Beginning negotiation
0621 05:10:39.371751 (+ 482us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:10:39.371999 (+ 248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:10:39.372002 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:10:39.372011 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:10:39.373097 (+ 1086us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0621 05:10:39.373097 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:10:39.373153 (+ 56us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0621 05:10:39.373206 (+ 53us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 05:10:39.373559 (+ 353us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 05:10:39.373567 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 05:10:39.373571 (+ 4us) client_negotiation.cc:769] Sending connection context
0621 05:10:39.373760 (+ 189us) client_negotiation.cc:240] Negotiation successful
0621 05:10:39.373865 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":98,"threads_started":1}
I20260621 05:10:39.374557 3598 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWZOx2_/build/release/bin/kudu
/tmp/dist-test-taskWZOx2_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782018616648578-3598-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782018616648578-3598-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782018616648578-3598-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782018616648578-3598-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.131.130:0
--local_ip_for_outbound_sockets=127.3.131.130
--webserver_interface=127.3.131.130
--webserver_port=0
--tserver_master_addrs=127.3.131.190:36105
--builtin_ntp_servers=127.3.131.148:44959
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260621 05:10:39.374755 8421 heartbeater.cc:347] Connected to a master server at 127.3.131.190:36105
I20260621 05:10:39.374848 8421 heartbeater.cc:464] Registering TS with master...
I20260621 05:10:39.374970 8421 heartbeater.cc:511] Master 127.3.131.190:36105 requested a full tablet report, sending...
I20260621 05:10:39.375797 8233 ts_manager.cc:194] Registered new tserver with Master: 3e5d2bacdfde4aba95becffd65fb03fb (127.3.131.129:43157)
W20260621 05:10:39.460362 8424 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 05:10:39.460590 8424 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 05:10:39.460616 8424 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 05:10:39.461938 8424 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 05:10:39.461978 8424 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 05:10:39.462005 8424 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.131.130
I20260621 05:10:39.463482 8424 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.131.148:44959
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782018616648578-3598-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782018616648578-3598-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.131.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782018616648578-3598-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.131.130
--webserver_port=0
--tserver_master_addrs=127.3.131.190:36105
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8424
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.131.130
--log_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782018616648578-3598-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 21 Jun 2026 04:16:16 UTC on ccycloud.quasar-xycaha.root.comops.site
build id 4457881
I20260621 05:10:39.463729 8424 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 05:10:39.464022 8424 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 05:10:39.469461 8429 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:10:39.469489 8432 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:10:39.469498 8430 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:10:39.469745 8424 server_base.cc:1034] running on GCE node
I20260621 05:10:39.470007 8424 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 05:10:39.470264 8424 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 05:10:39.471431 8424 hybrid_clock.cc:648] HybridClock initialized: now 1782018639471404 us; error 59 us; skew 500 ppm
I20260621 05:10:39.472851 8424 webserver.cc:460] Webserver started at http://127.3.131.130:38483/ using document root <none> and password file <none>
I20260621 05:10:39.473112 8424 fs_manager.cc:359] Metadata directory not provided
I20260621 05:10:39.473186 8424 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 05:10:39.473338 8424 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 05:10:39.474586 8424 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782018616648578-3598-0/minicluster-data/ts-1/data/instance:
uuid: "dcb5df66fb6a47cba61574c47f874330"
format_stamp: "Formatted at 2026-06-21 05:10:39 on dist-test-slave-vrq1"
server_key: "f3005ec96d6fa0eaf2364c6187f1c5bb"
server_key_iv: "16aec33fe85a60fb4d03d4e9ca721a12"
server_key_version: "encryptionkey@0"
I20260621 05:10:39.474965 8424 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782018616648578-3598-0/minicluster-data/ts-1/wal/instance:
uuid: "dcb5df66fb6a47cba61574c47f874330"
format_stamp: "Formatted at 2026-06-21 05:10:39 on dist-test-slave-vrq1"
server_key: "f3005ec96d6fa0eaf2364c6187f1c5bb"
server_key_iv: "16aec33fe85a60fb4d03d4e9ca721a12"
server_key_version: "encryptionkey@0"
I20260621 05:10:39.476416 8424 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260621 05:10:39.477365 8438 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 05:10:39.477689 8424 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260621 05:10:39.477799 8424 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782018616648578-3598-0/minicluster-data/ts-1/data,/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782018616648578-3598-0/minicluster-data/ts-1/wal
uuid: "dcb5df66fb6a47cba61574c47f874330"
format_stamp: "Formatted at 2026-06-21 05:10:39 on dist-test-slave-vrq1"
server_key: "f3005ec96d6fa0eaf2364c6187f1c5bb"
server_key_iv: "16aec33fe85a60fb4d03d4e9ca721a12"
server_key_version: "encryptionkey@0"
I20260621 05:10:39.477892 8424 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782018616648578-3598-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782018616648578-3598-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782018616648578-3598-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 05:10:39.481479 8424 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 05:10:39.481760 8424 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 05:10:39.481909 8424 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 05:10:39.482136 8424 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 05:10:39.482470 8424 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 05:10:39.482528 8424 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 05:10:39.482571 8424 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 05:10:39.482602 8424 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 05:10:39.489226 8424 rpc_server.cc:307] RPC server started. Bound to: 127.3.131.130:35439
I20260621 05:10:39.489262 8551 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.131.130:35439 every 8 connection(s)
I20260621 05:10:39.489635 8424 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782018616648578-3598-0/minicluster-data/ts-1/data/info.pb
I20260621 05:10:39.490888 3598 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWZOx2_/build/release/bin/kudu as pid 8424
I20260621 05:10:39.490993 3598 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782018616648578-3598-0/minicluster-data/ts-1/wal/instance
I20260621 05:10:39.491226 3598 external_mini_cluster.cc:1428] Setting key d92a74e347458ac0d81c664baddbef91
I20260621 05:10:39.492575 3598 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWZOx2_/build/release/bin/kudu
/tmp/dist-test-taskWZOx2_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782018616648578-3598-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782018616648578-3598-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782018616648578-3598-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782018616648578-3598-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.131.131:0
--local_ip_for_outbound_sockets=127.3.131.131
--webserver_interface=127.3.131.131
--webserver_port=0
--tserver_master_addrs=127.3.131.190:36105
--builtin_ntp_servers=127.3.131.148:44959
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260621 05:10:39.493773 8272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:10:39.490583 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.131.130:48679 (local address 127.3.131.190:36105)
0621 05:10:39.490654 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0621 05:10:39.490660 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 05:10:39.491000 (+ 340us) server_negotiation.cc:407] Connection header received
0621 05:10:39.491770 (+ 770us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:10:39.491772 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:10:39.491780 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:10:39.491816 (+ 36us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:10:39.493413 (+ 1597us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:10:39.493414 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:10:39.493415 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 05:10:39.493420 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:10:39.493443 (+ 23us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 05:10:39.493448 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:10:39.493504 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0621 05:10:39.493653 (+ 149us) server_negotiation.cc:299] Negotiation successful
0621 05:10:39.493670 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260621 05:10:39.494122 8554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:10:39.490579 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.131.190:36105 (local address 127.3.131.130:48679)
0621 05:10:39.490851 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0621 05:10:39.490897 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0621 05:10:39.491626 (+ 729us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:10:39.492024 (+ 398us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:10:39.492028 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:10:39.492039 (+ 11us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:10:39.493224 (+ 1185us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0621 05:10:39.493225 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:10:39.493267 (+ 42us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0621 05:10:39.493308 (+ 41us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 05:10:39.493532 (+ 224us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 05:10:39.493539 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 05:10:39.493542 (+ 3us) client_negotiation.cc:769] Sending connection context
0621 05:10:39.493661 (+ 119us) client_negotiation.cc:240] Negotiation successful
0621 05:10:39.493759 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":204,"thread_start_us":129,"threads_started":1}
I20260621 05:10:39.494653 8552 heartbeater.cc:347] Connected to a master server at 127.3.131.190:36105
I20260621 05:10:39.494753 8552 heartbeater.cc:464] Registering TS with master...
I20260621 05:10:39.494931 8552 heartbeater.cc:511] Master 127.3.131.190:36105 requested a full tablet report, sending...
I20260621 05:10:39.495409 8233 ts_manager.cc:194] Registered new tserver with Master: dcb5df66fb6a47cba61574c47f874330 (127.3.131.130:35439)
W20260621 05:10:39.580753 8555 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 05:10:39.580962 8555 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 05:10:39.580986 8555 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 05:10:39.582242 8555 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 05:10:39.582286 8555 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 05:10:39.582314 8555 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.131.131
I20260621 05:10:39.583681 8555 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.131.148:44959
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782018616648578-3598-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782018616648578-3598-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.131.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782018616648578-3598-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.131.131
--webserver_port=0
--tserver_master_addrs=127.3.131.190:36105
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8555
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.131.131
--log_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782018616648578-3598-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 21 Jun 2026 04:16:16 UTC on ccycloud.quasar-xycaha.root.comops.site
build id 4457881
I20260621 05:10:39.583889 8555 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 05:10:39.584141 8555 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 05:10:39.589655 8560 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:10:39.589748 8561 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:10:39.589684 8563 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:10:39.590991 8555 server_base.cc:1034] running on GCE node
I20260621 05:10:39.591272 8555 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 05:10:39.591507 8555 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 05:10:39.592676 8555 hybrid_clock.cc:648] HybridClock initialized: now 1782018639592665 us; error 38 us; skew 500 ppm
I20260621 05:10:39.594271 8555 webserver.cc:460] Webserver started at http://127.3.131.131:43899/ using document root <none> and password file <none>
I20260621 05:10:39.594535 8555 fs_manager.cc:359] Metadata directory not provided
I20260621 05:10:39.594585 8555 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 05:10:39.594722 8555 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 05:10:39.595950 8555 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782018616648578-3598-0/minicluster-data/ts-2/data/instance:
uuid: "76ee766245ac40bfaa950ec259907c51"
format_stamp: "Formatted at 2026-06-21 05:10:39 on dist-test-slave-vrq1"
server_key: "9634de619bef29b288b30e8ddb40875c"
server_key_iv: "26352b7a654173ae40e67219060dcfae"
server_key_version: "encryptionkey@0"
I20260621 05:10:39.596323 8555 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782018616648578-3598-0/minicluster-data/ts-2/wal/instance:
uuid: "76ee766245ac40bfaa950ec259907c51"
format_stamp: "Formatted at 2026-06-21 05:10:39 on dist-test-slave-vrq1"
server_key: "9634de619bef29b288b30e8ddb40875c"
server_key_iv: "26352b7a654173ae40e67219060dcfae"
server_key_version: "encryptionkey@0"
I20260621 05:10:39.597728 8555 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260621 05:10:39.598574 8569 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 05:10:39.598843 8555 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260621 05:10:39.598958 8555 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782018616648578-3598-0/minicluster-data/ts-2/data,/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782018616648578-3598-0/minicluster-data/ts-2/wal
uuid: "76ee766245ac40bfaa950ec259907c51"
format_stamp: "Formatted at 2026-06-21 05:10:39 on dist-test-slave-vrq1"
server_key: "9634de619bef29b288b30e8ddb40875c"
server_key_iv: "26352b7a654173ae40e67219060dcfae"
server_key_version: "encryptionkey@0"
I20260621 05:10:39.599045 8555 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782018616648578-3598-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782018616648578-3598-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782018616648578-3598-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 05:10:39.602762 8555 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 05:10:39.603040 8555 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 05:10:39.603221 8555 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 05:10:39.603444 8555 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 05:10:39.603757 8555 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 05:10:39.603811 8555 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 05:10:39.603849 8555 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 05:10:39.603875 8555 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 05:10:39.610019 8555 rpc_server.cc:307] RPC server started. Bound to: 127.3.131.131:35235
I20260621 05:10:39.610091 8682 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.131.131:35235 every 8 connection(s)
I20260621 05:10:39.610438 8555 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782018616648578-3598-0/minicluster-data/ts-2/data/info.pb
I20260621 05:10:39.613757 8272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:10:39.611377 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.131.131:45349 (local address 127.3.131.190:36105)
0621 05:10:39.611463 (+ 86us) server_negotiation.cc:206] Beginning negotiation
0621 05:10:39.611465 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:10:39.611724 (+ 259us) server_negotiation.cc:407] Connection header received
0621 05:10:39.612264 (+ 540us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:10:39.612266 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:10:39.612270 (+ 4us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:10:39.612362 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:10:39.613524 (+ 1162us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:10:39.613525 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:10:39.613525 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 05:10:39.613529 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:10:39.613544 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 05:10:39.613549 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:10:39.613613 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0621 05:10:39.613680 (+ 67us) server_negotiation.cc:299] Negotiation successful
0621 05:10:39.613694 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260621 05:10:39.613910 8685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:10:39.611384 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.131.190:36105 (local address 127.3.131.131:45349)
0621 05:10:39.611603 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0621 05:10:39.611636 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0621 05:10:39.612111 (+ 475us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:10:39.612421 (+ 310us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:10:39.612424 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:10:39.612433 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:10:39.613373 (+ 940us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0621 05:10:39.613373 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:10:39.613401 (+ 28us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0621 05:10:39.613431 (+ 30us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 05:10:39.613613 (+ 182us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 05:10:39.613616 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 05:10:39.613618 (+ 2us) client_negotiation.cc:769] Sending connection context
0621 05:10:39.613672 (+ 54us) client_negotiation.cc:240] Negotiation successful
0621 05:10:39.613721 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":157,"thread_start_us":107,"threads_started":1}
I20260621 05:10:39.614590 8683 heartbeater.cc:347] Connected to a master server at 127.3.131.190:36105
I20260621 05:10:39.614709 8683 heartbeater.cc:464] Registering TS with master...
I20260621 05:10:39.614866 8683 heartbeater.cc:511] Master 127.3.131.190:36105 requested a full tablet report, sending...
I20260621 05:10:39.615397 8233 ts_manager.cc:194] Registered new tserver with Master: 76ee766245ac40bfaa950ec259907c51 (127.3.131.131:35235)
I20260621 05:10:39.618973 3598 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWZOx2_/build/release/bin/kudu as pid 8555
I20260621 05:10:39.619056 3598 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782018616648578-3598-0/minicluster-data/ts-2/wal/instance
I20260621 05:10:39.619299 3598 external_mini_cluster.cc:1428] Setting key bc1ef44bb1c50398a29924a7f16aad76
I20260621 05:10:39.620342 3598 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260621 05:10:39.622988 8272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:10:39.621893 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50668 (local address 127.3.131.190:36105)
0621 05:10:39.622002 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0621 05:10:39.622003 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0621 05:10:39.622027 (+ 24us) server_negotiation.cc:407] Connection header received
0621 05:10:39.622110 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:10:39.622111 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:10:39.622115 (+ 4us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:10:39.622167 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:10:39.622722 (+ 555us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:10:39.622723 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:10:39.622723 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 05:10:39.622726 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:10:39.622740 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 05:10:39.622743 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:10:39.622788 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0621 05:10:39.622856 (+ 68us) server_negotiation.cc:299] Negotiation successful
0621 05:10:39.622868 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260621 05:10:39.624830 8233 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50668:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260621 05:10:39.626363 8233 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260621 05:10:39.630730 8691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:10:39.628764 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.131.129:43157 (local address 127.0.0.1:36442)
0621 05:10:39.629072 (+ 308us) negotiation.cc:107] Waiting for socket to connect
0621 05:10:39.629115 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0621 05:10:39.629210 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:10:39.629376 (+ 166us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:10:39.629381 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:10:39.629388 (+ 7us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:10:39.630237 (+ 849us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0621 05:10:39.630238 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:10:39.630254 (+ 16us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0621 05:10:39.630282 (+ 28us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 05:10:39.630546 (+ 264us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 05:10:39.630553 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 05:10:39.630556 (+ 3us) client_negotiation.cc:769] Sending connection context
0621 05:10:39.630625 (+ 69us) client_negotiation.cc:240] Negotiation successful
0621 05:10:39.630641 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":283,"thread_start_us":72,"threads_started":1}
I20260621 05:10:39.630754 8694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:10:39.628850 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36442 (local address 127.3.131.129:43157)
0621 05:10:39.629047 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0621 05:10:39.629052 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 05:10:39.629182 (+ 130us) server_negotiation.cc:407] Connection header received
0621 05:10:39.629280 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:10:39.629284 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:10:39.629293 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:10:39.629366 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:10:39.630379 (+ 1013us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:10:39.630388 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:10:39.630389 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 05:10:39.630393 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:10:39.630432 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 05:10:39.630460 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:10:39.630519 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0621 05:10:39.630646 (+ 127us) server_negotiation.cc:299] Negotiation successful
0621 05:10:39.630667 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":158,"thread_start_us":119,"threads_started":1}
I20260621 05:10:39.630824 8692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:10:39.628771 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.131.131:35235 (local address 127.0.0.1:50754)
0621 05:10:39.629247 (+ 476us) negotiation.cc:107] Waiting for socket to connect
0621 05:10:39.629257 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0621 05:10:39.629339 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:10:39.629795 (+ 456us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:10:39.629796 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:10:39.629799 (+ 3us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:10:39.630406 (+ 607us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0621 05:10:39.630407 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:10:39.630414 (+ 7us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0621 05:10:39.630427 (+ 13us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 05:10:39.630708 (+ 281us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 05:10:39.630708 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 05:10:39.630709 (+ 1us) client_negotiation.cc:769] Sending connection context
0621 05:10:39.630749 (+ 40us) client_negotiation.cc:240] Negotiation successful
0621 05:10:39.630760 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":458,"spinlock_wait_cycles":768,"thread_start_us":98,"threads_started":1}
I20260621 05:10:39.631069 8696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:10:39.629007 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50754 (local address 127.3.131.131:35235)
0621 05:10:39.629229 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0621 05:10:39.629233 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 05:10:39.629340 (+ 107us) server_negotiation.cc:407] Connection header received
0621 05:10:39.629403 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:10:39.629407 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:10:39.629417 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:10:39.629488 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:10:39.630555 (+ 1067us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:10:39.630564 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:10:39.630564 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 05:10:39.630570 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:10:39.630603 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 05:10:39.630631 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:10:39.630684 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0621 05:10:39.630907 (+ 223us) server_negotiation.cc:299] Negotiation successful
0621 05:10:39.630928 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":185,"thread_start_us":147,"threads_started":1}
I20260621 05:10:39.631246 8693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:10:39.628772 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.131.130:35439 (local address 127.0.0.1:43356)
0621 05:10:39.629421 (+ 649us) negotiation.cc:107] Waiting for socket to connect
0621 05:10:39.629430 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0621 05:10:39.629470 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:10:39.629877 (+ 407us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:10:39.629878 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:10:39.629880 (+ 2us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:10:39.630588 (+ 708us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0621 05:10:39.630589 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:10:39.630594 (+ 5us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0621 05:10:39.630605 (+ 11us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 05:10:39.631086 (+ 481us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 05:10:39.631087 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 05:10:39.631088 (+ 1us) client_negotiation.cc:769] Sending connection context
0621 05:10:39.631145 (+ 57us) client_negotiation.cc:240] Negotiation successful
0621 05:10:39.631160 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":635,"mutex_wait_us":2,"thread_start_us":102,"threads_started":1}
I20260621 05:10:39.631441 8695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:10:39.629009 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43356 (local address 127.3.131.130:35439)
0621 05:10:39.629166 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0621 05:10:39.629170 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 05:10:39.629524 (+ 354us) server_negotiation.cc:407] Connection header received
0621 05:10:39.629552 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:10:39.629555 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:10:39.629563 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:10:39.629629 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:10:39.630694 (+ 1065us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:10:39.630700 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:10:39.630701 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 05:10:39.630705 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:10:39.630738 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 05:10:39.630766 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:10:39.631040 (+ 274us) server_negotiation.cc:1035] Waiting for connection context
0621 05:10:39.631345 (+ 305us) server_negotiation.cc:299] Negotiation successful
0621 05:10:39.631363 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":116,"thread_start_us":80,"threads_started":1}
I20260621 05:10:39.631733 8355 tablet_service.cc:1467] Processing CreateTablet for tablet 561c27939bc14a88b823ee357a52bded (DEFAULT_TABLE table=test-table [id=dfafba98da6e4082b91f606c57db05af]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 05:10:39.631731 8617 tablet_service.cc:1467] Processing CreateTablet for tablet 561c27939bc14a88b823ee357a52bded (DEFAULT_TABLE table=test-table [id=dfafba98da6e4082b91f606c57db05af]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 05:10:39.632035 8355 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 561c27939bc14a88b823ee357a52bded. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 05:10:39.632073 8617 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 561c27939bc14a88b823ee357a52bded. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 05:10:39.632062 8486 tablet_service.cc:1467] Processing CreateTablet for tablet 561c27939bc14a88b823ee357a52bded (DEFAULT_TABLE table=test-table [id=dfafba98da6e4082b91f606c57db05af]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 05:10:39.632380 8486 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 561c27939bc14a88b823ee357a52bded. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 05:10:39.634426 8697 tablet_bootstrap.cc:492] T 561c27939bc14a88b823ee357a52bded P 3e5d2bacdfde4aba95becffd65fb03fb: Bootstrap starting.
I20260621 05:10:39.635128 8697 tablet_bootstrap.cc:654] T 561c27939bc14a88b823ee357a52bded P 3e5d2bacdfde4aba95becffd65fb03fb: Neither blocks nor log segments found. Creating new log.
I20260621 05:10:39.635470 8697 log.cc:824] T 561c27939bc14a88b823ee357a52bded P 3e5d2bacdfde4aba95becffd65fb03fb: Log is configured to *not* fsync() on all Append() calls
I20260621 05:10:39.635483 8698 tablet_bootstrap.cc:492] T 561c27939bc14a88b823ee357a52bded P 76ee766245ac40bfaa950ec259907c51: Bootstrap starting.
I20260621 05:10:39.635602 8699 tablet_bootstrap.cc:492] T 561c27939bc14a88b823ee357a52bded P dcb5df66fb6a47cba61574c47f874330: Bootstrap starting.
I20260621 05:10:39.636211 8698 tablet_bootstrap.cc:654] T 561c27939bc14a88b823ee357a52bded P 76ee766245ac40bfaa950ec259907c51: Neither blocks nor log segments found. Creating new log.
I20260621 05:10:39.636287 8699 tablet_bootstrap.cc:654] T 561c27939bc14a88b823ee357a52bded P dcb5df66fb6a47cba61574c47f874330: Neither blocks nor log segments found. Creating new log.
I20260621 05:10:39.636301 8697 tablet_bootstrap.cc:492] T 561c27939bc14a88b823ee357a52bded P 3e5d2bacdfde4aba95becffd65fb03fb: No bootstrap required, opened a new log
I20260621 05:10:39.636358 8697 ts_tablet_manager.cc:1397] T 561c27939bc14a88b823ee357a52bded P 3e5d2bacdfde4aba95becffd65fb03fb: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260621 05:10:39.636425 8698 log.cc:824] T 561c27939bc14a88b823ee357a52bded P 76ee766245ac40bfaa950ec259907c51: Log is configured to *not* fsync() on all Append() calls
I20260621 05:10:39.636595 8699 log.cc:824] T 561c27939bc14a88b823ee357a52bded P dcb5df66fb6a47cba61574c47f874330: Log is configured to *not* fsync() on all Append() calls
I20260621 05:10:39.637120 8698 tablet_bootstrap.cc:492] T 561c27939bc14a88b823ee357a52bded P 76ee766245ac40bfaa950ec259907c51: No bootstrap required, opened a new log
I20260621 05:10:39.637178 8698 ts_tablet_manager.cc:1397] T 561c27939bc14a88b823ee357a52bded P 76ee766245ac40bfaa950ec259907c51: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260621 05:10:39.638311 8697 raft_consensus.cc:348] T 561c27939bc14a88b823ee357a52bded P 3e5d2bacdfde4aba95becffd65fb03fb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcb5df66fb6a47cba61574c47f874330" member_type: VOTER last_known_addr { host: "127.3.131.130" port: 35439 } } peers { permanent_uuid: "76ee766245ac40bfaa950ec259907c51" member_type: VOTER last_known_addr { host: "127.3.131.131" port: 35235 } } peers { permanent_uuid: "3e5d2bacdfde4aba95becffd65fb03fb" member_type: VOTER last_known_addr { host: "127.3.131.129" port: 43157 } }
I20260621 05:10:39.638434 8697 raft_consensus.cc:374] T 561c27939bc14a88b823ee357a52bded P 3e5d2bacdfde4aba95becffd65fb03fb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 05:10:39.638454 8697 raft_consensus.cc:729] T 561c27939bc14a88b823ee357a52bded P 3e5d2bacdfde4aba95becffd65fb03fb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3e5d2bacdfde4aba95becffd65fb03fb, State: Initialized, Role: FOLLOWER
I20260621 05:10:39.638590 8697 consensus_queue.cc:260] T 561c27939bc14a88b823ee357a52bded P 3e5d2bacdfde4aba95becffd65fb03fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcb5df66fb6a47cba61574c47f874330" member_type: VOTER last_known_addr { host: "127.3.131.130" port: 35439 } } peers { permanent_uuid: "76ee766245ac40bfaa950ec259907c51" member_type: VOTER last_known_addr { host: "127.3.131.131" port: 35235 } } peers { permanent_uuid: "3e5d2bacdfde4aba95becffd65fb03fb" member_type: VOTER last_known_addr { host: "127.3.131.129" port: 43157 } }
I20260621 05:10:39.638810 8697 ts_tablet_manager.cc:1428] T 561c27939bc14a88b823ee357a52bded P 3e5d2bacdfde4aba95becffd65fb03fb: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260621 05:10:39.638958 8421 heartbeater.cc:503] Master 127.3.131.190:36105 was elected leader, sending a full tablet report...
I20260621 05:10:39.639099 8698 raft_consensus.cc:348] T 561c27939bc14a88b823ee357a52bded P 76ee766245ac40bfaa950ec259907c51 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcb5df66fb6a47cba61574c47f874330" member_type: VOTER last_known_addr { host: "127.3.131.130" port: 35439 } } peers { permanent_uuid: "76ee766245ac40bfaa950ec259907c51" member_type: VOTER last_known_addr { host: "127.3.131.131" port: 35235 } } peers { permanent_uuid: "3e5d2bacdfde4aba95becffd65fb03fb" member_type: VOTER last_known_addr { host: "127.3.131.129" port: 43157 } }
I20260621 05:10:39.639281 8698 raft_consensus.cc:374] T 561c27939bc14a88b823ee357a52bded P 76ee766245ac40bfaa950ec259907c51 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 05:10:39.639325 8698 raft_consensus.cc:729] T 561c27939bc14a88b823ee357a52bded P 76ee766245ac40bfaa950ec259907c51 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 76ee766245ac40bfaa950ec259907c51, State: Initialized, Role: FOLLOWER
I20260621 05:10:39.639431 8698 consensus_queue.cc:260] T 561c27939bc14a88b823ee357a52bded P 76ee766245ac40bfaa950ec259907c51 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcb5df66fb6a47cba61574c47f874330" member_type: VOTER last_known_addr { host: "127.3.131.130" port: 35439 } } peers { permanent_uuid: "76ee766245ac40bfaa950ec259907c51" member_type: VOTER last_known_addr { host: "127.3.131.131" port: 35235 } } peers { permanent_uuid: "3e5d2bacdfde4aba95becffd65fb03fb" member_type: VOTER last_known_addr { host: "127.3.131.129" port: 43157 } }
I20260621 05:10:39.639717 8683 heartbeater.cc:503] Master 127.3.131.190:36105 was elected leader, sending a full tablet report...
I20260621 05:10:39.639791 8699 tablet_bootstrap.cc:492] T 561c27939bc14a88b823ee357a52bded P dcb5df66fb6a47cba61574c47f874330: No bootstrap required, opened a new log
I20260621 05:10:39.639858 8699 ts_tablet_manager.cc:1397] T 561c27939bc14a88b823ee357a52bded P dcb5df66fb6a47cba61574c47f874330: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20260621 05:10:39.639709 8698 ts_tablet_manager.cc:1428] T 561c27939bc14a88b823ee357a52bded P 76ee766245ac40bfaa950ec259907c51: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260621 05:10:39.642061 8699 raft_consensus.cc:348] T 561c27939bc14a88b823ee357a52bded P dcb5df66fb6a47cba61574c47f874330 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcb5df66fb6a47cba61574c47f874330" member_type: VOTER last_known_addr { host: "127.3.131.130" port: 35439 } } peers { permanent_uuid: "76ee766245ac40bfaa950ec259907c51" member_type: VOTER last_known_addr { host: "127.3.131.131" port: 35235 } } peers { permanent_uuid: "3e5d2bacdfde4aba95becffd65fb03fb" member_type: VOTER last_known_addr { host: "127.3.131.129" port: 43157 } }
I20260621 05:10:39.642210 8699 raft_consensus.cc:374] T 561c27939bc14a88b823ee357a52bded P dcb5df66fb6a47cba61574c47f874330 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 05:10:39.642230 8699 raft_consensus.cc:729] T 561c27939bc14a88b823ee357a52bded P dcb5df66fb6a47cba61574c47f874330 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dcb5df66fb6a47cba61574c47f874330, State: Initialized, Role: FOLLOWER
I20260621 05:10:39.642350 8699 consensus_queue.cc:260] T 561c27939bc14a88b823ee357a52bded P dcb5df66fb6a47cba61574c47f874330 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcb5df66fb6a47cba61574c47f874330" member_type: VOTER last_known_addr { host: "127.3.131.130" port: 35439 } } peers { permanent_uuid: "76ee766245ac40bfaa950ec259907c51" member_type: VOTER last_known_addr { host: "127.3.131.131" port: 35235 } } peers { permanent_uuid: "3e5d2bacdfde4aba95becffd65fb03fb" member_type: VOTER last_known_addr { host: "127.3.131.129" port: 43157 } }
I20260621 05:10:39.642596 8699 ts_tablet_manager.cc:1428] T 561c27939bc14a88b823ee357a52bded P dcb5df66fb6a47cba61574c47f874330: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260621 05:10:39.642683 8552 heartbeater.cc:503] Master 127.3.131.190:36105 was elected leader, sending a full tablet report...
W20260621 05:10:39.740293 8553 tablet.cc:2369] T 561c27939bc14a88b823ee357a52bded P dcb5df66fb6a47cba61574c47f874330: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260621 05:10:39.812611 8703 raft_consensus.cc:482] T 561c27939bc14a88b823ee357a52bded P 3e5d2bacdfde4aba95becffd65fb03fb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260621 05:10:39.812757 8703 raft_consensus.cc:504] T 561c27939bc14a88b823ee357a52bded P 3e5d2bacdfde4aba95becffd65fb03fb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcb5df66fb6a47cba61574c47f874330" member_type: VOTER last_known_addr { host: "127.3.131.130" port: 35439 } } peers { permanent_uuid: "76ee766245ac40bfaa950ec259907c51" member_type: VOTER last_known_addr { host: "127.3.131.131" port: 35235 } } peers { permanent_uuid: "3e5d2bacdfde4aba95becffd65fb03fb" member_type: VOTER last_known_addr { host: "127.3.131.129" port: 43157 } }
I20260621 05:10:39.813510 8703 leader_election.cc:288] T 561c27939bc14a88b823ee357a52bded P 3e5d2bacdfde4aba95becffd65fb03fb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dcb5df66fb6a47cba61574c47f874330 (127.3.131.130:35439), 76ee766245ac40bfaa950ec259907c51 (127.3.131.131:35235)
I20260621 05:10:39.815312 8695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:10:39.813969 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.131.129:41631 (local address 127.3.131.130:35439)
0621 05:10:39.814051 (+ 82us) server_negotiation.cc:206] Beginning negotiation
0621 05:10:39.814054 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0621 05:10:39.814201 (+ 147us) server_negotiation.cc:407] Connection header received
0621 05:10:39.814235 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:10:39.814236 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:10:39.814243 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:10:39.814297 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:10:39.815034 (+ 737us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:10:39.815036 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:10:39.815036 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 05:10:39.815041 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:10:39.815059 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 05:10:39.815063 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:10:39.815120 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0621 05:10:39.815220 (+ 100us) server_negotiation.cc:299] Negotiation successful
0621 05:10:39.815237 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260621 05:10:39.815320 8423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:10:39.813745 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.131.131:35235 (local address 127.3.131.129:48699)
0621 05:10:39.813806 (+ 61us) negotiation.cc:107] Waiting for socket to connect
0621 05:10:39.813819 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0621 05:10:39.813869 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:10:39.814234 (+ 365us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:10:39.814235 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:10:39.814237 (+ 2us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:10:39.814790 (+ 553us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0621 05:10:39.814791 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:10:39.814823 (+ 32us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0621 05:10:39.814839 (+ 16us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 05:10:39.815072 (+ 233us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 05:10:39.815072 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 05:10:39.815073 (+ 1us) client_negotiation.cc:769] Sending connection context
0621 05:10:39.815210 (+ 137us) client_negotiation.cc:240] Negotiation successful
0621 05:10:39.815228 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":37}
I20260621 05:10:39.815332 8706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:10:39.813806 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.131.130:35439 (local address 127.3.131.129:41631)
0621 05:10:39.814071 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0621 05:10:39.814092 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0621 05:10:39.814145 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:10:39.814312 (+ 167us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:10:39.814313 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:10:39.814314 (+ 1us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:10:39.814921 (+ 607us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0621 05:10:39.814921 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:10:39.814934 (+ 13us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0621 05:10:39.814942 (+ 8us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 05:10:39.815130 (+ 188us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 05:10:39.815131 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 05:10:39.815131 (+ 0us) client_negotiation.cc:769] Sending connection context
0621 05:10:39.815219 (+ 88us) client_negotiation.cc:240] Negotiation successful
0621 05:10:39.815230 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":248,"thread_start_us":120,"threads_started":1}
I20260621 05:10:39.815361 8696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:10:39.813993 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.131.129:48699 (local address 127.3.131.131:35235)
0621 05:10:39.814037 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0621 05:10:39.814064 (+ 27us) server_negotiation.cc:399] Waiting for connection header
0621 05:10:39.814073 (+ 9us) server_negotiation.cc:407] Connection header received
0621 05:10:39.814106 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:10:39.814107 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:10:39.814113 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:10:39.814218 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:10:39.814942 (+ 724us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:10:39.814943 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:10:39.814943 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 05:10:39.814948 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:10:39.814965 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 05:10:39.815005 (+ 40us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:10:39.815065 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0621 05:10:39.815244 (+ 179us) server_negotiation.cc:299] Negotiation successful
0621 05:10:39.815258 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260621 05:10:39.815759 8506 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "561c27939bc14a88b823ee357a52bded" candidate_uuid: "3e5d2bacdfde4aba95becffd65fb03fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dcb5df66fb6a47cba61574c47f874330" is_pre_election: true
I20260621 05:10:39.815842 8637 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "561c27939bc14a88b823ee357a52bded" candidate_uuid: "3e5d2bacdfde4aba95becffd65fb03fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76ee766245ac40bfaa950ec259907c51" is_pre_election: true
I20260621 05:10:39.815953 8506 raft_consensus.cc:2454] T 561c27939bc14a88b823ee357a52bded P dcb5df66fb6a47cba61574c47f874330 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3e5d2bacdfde4aba95becffd65fb03fb in term 0.
I20260621 05:10:39.815963 8637 raft_consensus.cc:2454] T 561c27939bc14a88b823ee357a52bded P 76ee766245ac40bfaa950ec259907c51 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3e5d2bacdfde4aba95becffd65fb03fb in term 0.
I20260621 05:10:39.816254 8308 leader_election.cc:302] T 561c27939bc14a88b823ee357a52bded P 3e5d2bacdfde4aba95becffd65fb03fb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3e5d2bacdfde4aba95becffd65fb03fb, dcb5df66fb6a47cba61574c47f874330; no voters:
I20260621 05:10:39.816416 8703 raft_consensus.cc:2781] T 561c27939bc14a88b823ee357a52bded P 3e5d2bacdfde4aba95becffd65fb03fb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260621 05:10:39.816476 8703 raft_consensus.cc:482] T 561c27939bc14a88b823ee357a52bded P 3e5d2bacdfde4aba95becffd65fb03fb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260621 05:10:39.816501 8703 raft_consensus.cc:3037] T 561c27939bc14a88b823ee357a52bded P 3e5d2bacdfde4aba95becffd65fb03fb [term 0 FOLLOWER]: Advancing to term 1
I20260621 05:10:39.817405 8703 raft_consensus.cc:504] T 561c27939bc14a88b823ee357a52bded P 3e5d2bacdfde4aba95becffd65fb03fb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcb5df66fb6a47cba61574c47f874330" member_type: VOTER last_known_addr { host: "127.3.131.130" port: 35439 } } peers { permanent_uuid: "76ee766245ac40bfaa950ec259907c51" member_type: VOTER last_known_addr { host: "127.3.131.131" port: 35235 } } peers { permanent_uuid: "3e5d2bacdfde4aba95becffd65fb03fb" member_type: VOTER last_known_addr { host: "127.3.131.129" port: 43157 } }
I20260621 05:10:39.817555 8703 leader_election.cc:288] T 561c27939bc14a88b823ee357a52bded P 3e5d2bacdfde4aba95becffd65fb03fb [CANDIDATE]: Term 1 election: Requested vote from peers dcb5df66fb6a47cba61574c47f874330 (127.3.131.130:35439), 76ee766245ac40bfaa950ec259907c51 (127.3.131.131:35235)
I20260621 05:10:39.817772 8637 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "561c27939bc14a88b823ee357a52bded" candidate_uuid: "3e5d2bacdfde4aba95becffd65fb03fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76ee766245ac40bfaa950ec259907c51"
I20260621 05:10:39.817762 8506 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "561c27939bc14a88b823ee357a52bded" candidate_uuid: "3e5d2bacdfde4aba95becffd65fb03fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dcb5df66fb6a47cba61574c47f874330"
I20260621 05:10:39.817847 8637 raft_consensus.cc:3037] T 561c27939bc14a88b823ee357a52bded P 76ee766245ac40bfaa950ec259907c51 [term 0 FOLLOWER]: Advancing to term 1
I20260621 05:10:39.817878 8506 raft_consensus.cc:3037] T 561c27939bc14a88b823ee357a52bded P dcb5df66fb6a47cba61574c47f874330 [term 0 FOLLOWER]: Advancing to term 1
I20260621 05:10:39.818584 8637 raft_consensus.cc:2454] T 561c27939bc14a88b823ee357a52bded P 76ee766245ac40bfaa950ec259907c51 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3e5d2bacdfde4aba95becffd65fb03fb in term 1.
I20260621 05:10:39.818739 8506 raft_consensus.cc:2454] T 561c27939bc14a88b823ee357a52bded P dcb5df66fb6a47cba61574c47f874330 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3e5d2bacdfde4aba95becffd65fb03fb in term 1.
I20260621 05:10:39.818782 8309 leader_election.cc:302] T 561c27939bc14a88b823ee357a52bded P 3e5d2bacdfde4aba95becffd65fb03fb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3e5d2bacdfde4aba95becffd65fb03fb, 76ee766245ac40bfaa950ec259907c51; no voters:
I20260621 05:10:39.818888 8703 raft_consensus.cc:2781] T 561c27939bc14a88b823ee357a52bded P 3e5d2bacdfde4aba95becffd65fb03fb [term 1 FOLLOWER]: Leader election won for term 1
I20260621 05:10:39.819026 8703 raft_consensus.cc:686] T 561c27939bc14a88b823ee357a52bded P 3e5d2bacdfde4aba95becffd65fb03fb [term 1 LEADER]: Becoming Leader. State: Replica: 3e5d2bacdfde4aba95becffd65fb03fb, State: Running, Role: LEADER
I20260621 05:10:39.819155 8703 consensus_queue.cc:237] T 561c27939bc14a88b823ee357a52bded P 3e5d2bacdfde4aba95becffd65fb03fb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcb5df66fb6a47cba61574c47f874330" member_type: VOTER last_known_addr { host: "127.3.131.130" port: 35439 } } peers { permanent_uuid: "76ee766245ac40bfaa950ec259907c51" member_type: VOTER last_known_addr { host: "127.3.131.131" port: 35235 } } peers { permanent_uuid: "3e5d2bacdfde4aba95becffd65fb03fb" member_type: VOTER last_known_addr { host: "127.3.131.129" port: 43157 } }
I20260621 05:10:39.820209 8232 catalog_manager.cc:5495] T 561c27939bc14a88b823ee357a52bded P 3e5d2bacdfde4aba95becffd65fb03fb reported cstate change: term changed from 0 to 1, leader changed from <none> to 3e5d2bacdfde4aba95becffd65fb03fb (127.3.131.129). New cstate: current_term: 1 leader_uuid: "3e5d2bacdfde4aba95becffd65fb03fb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcb5df66fb6a47cba61574c47f874330" member_type: VOTER last_known_addr { host: "127.3.131.130" port: 35439 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "76ee766245ac40bfaa950ec259907c51" member_type: VOTER last_known_addr { host: "127.3.131.131" port: 35235 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3e5d2bacdfde4aba95becffd65fb03fb" member_type: VOTER last_known_addr { host: "127.3.131.129" port: 43157 } health_report { overall_health: HEALTHY } } }
W20260621 05:10:39.861231 8684 tablet.cc:2369] T 561c27939bc14a88b823ee357a52bded P 76ee766245ac40bfaa950ec259907c51: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260621 05:10:39.863336 8694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:10:39.862247 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36454 (local address 127.3.131.129:43157)
0621 05:10:39.862318 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0621 05:10:39.862320 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:10:39.862328 (+ 8us) server_negotiation.cc:407] Connection header received
0621 05:10:39.862374 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:10:39.862375 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:10:39.862387 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:10:39.862452 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:10:39.863080 (+ 628us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:10:39.863081 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:10:39.863081 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 05:10:39.863086 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:10:39.863102 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 05:10:39.863106 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:10:39.863160 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0621 05:10:39.863244 (+ 84us) server_negotiation.cc:299] Negotiation successful
0621 05:10:39.863259 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260621 05:10:39.865422 8637 raft_consensus.cc:1261] T 561c27939bc14a88b823ee357a52bded P 76ee766245ac40bfaa950ec259907c51 [term 1 FOLLOWER]: Refusing update from remote peer 3e5d2bacdfde4aba95becffd65fb03fb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 05:10:39.865563 8506 raft_consensus.cc:1261] T 561c27939bc14a88b823ee357a52bded P dcb5df66fb6a47cba61574c47f874330 [term 1 FOLLOWER]: Refusing update from remote peer 3e5d2bacdfde4aba95becffd65fb03fb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 05:10:39.865828 8703 consensus_queue.cc:1035] T 561c27939bc14a88b823ee357a52bded P 3e5d2bacdfde4aba95becffd65fb03fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "76ee766245ac40bfaa950ec259907c51" member_type: VOTER last_known_addr { host: "127.3.131.131" port: 35235 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 05:10:39.866012 8707 consensus_queue.cc:1035] T 561c27939bc14a88b823ee357a52bded P 3e5d2bacdfde4aba95becffd65fb03fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "dcb5df66fb6a47cba61574c47f874330" member_type: VOTER last_known_addr { host: "127.3.131.130" port: 35439 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 05:10:39.868476 8711 mvcc.cc:205] Tried to move back new op lower bound from 7299148348885262336 to 7299148348699848704. Current Snapshot: MvccSnapshot[applied={T|T < 7299148348885262336}]
I20260621 05:10:39.868620 8712 mvcc.cc:205] Tried to move back new op lower bound from 7299148348885262336 to 7299148348699848704. Current Snapshot: MvccSnapshot[applied={T|T < 7299148348885262336}]
I20260621 05:10:39.870901 8232 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:50668:
table { table_name: "test-table" } modify_external_catalogs: true
I20260621 05:10:39.871033 8232 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:50668:
table { table_name: "test-table" } modify_external_catalogs: true
I20260621 05:10:39.871845 8232 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 7f7b32217bc84f5c938dbd1da1cc85a4: Sending DeleteTablet for 3 replicas of tablet 561c27939bc14a88b823ee357a52bded
I20260621 05:10:39.872191 8486 tablet_service.cc:1514] Processing DeleteTablet for tablet 561c27939bc14a88b823ee357a52bded with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 05:10:39 UTC) from {username='slave'} at 127.0.0.1:43356
I20260621 05:10:39.872215 8617 tablet_service.cc:1514] Processing DeleteTablet for tablet 561c27939bc14a88b823ee357a52bded with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 05:10:39 UTC) from {username='slave'} at 127.0.0.1:50754
I20260621 05:10:39.872308 8355 tablet_service.cc:1514] Processing DeleteTablet for tablet 561c27939bc14a88b823ee357a52bded with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 05:10:39 UTC) from {username='slave'} at 127.0.0.1:36442
I20260621 05:10:39.872466 8721 tablet_replica.cc:330] T 561c27939bc14a88b823ee357a52bded P dcb5df66fb6a47cba61574c47f874330: stopping tablet replica
I20260621 05:10:39.872576 8721 raft_consensus.cc:2229] T 561c27939bc14a88b823ee357a52bded P dcb5df66fb6a47cba61574c47f874330 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260621 05:10:39.872687 8721 raft_consensus.cc:2258] T 561c27939bc14a88b823ee357a52bded P dcb5df66fb6a47cba61574c47f874330 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 05:10:39.872808 3598 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWZOx2_/build/release/bin/kudu with pid 8293
I20260621 05:10:39.872915 8720 tablet_replica.cc:330] T 561c27939bc14a88b823ee357a52bded P 76ee766245ac40bfaa950ec259907c51: stopping tablet replica
I20260621 05:10:39.873019 8720 raft_consensus.cc:2229] T 561c27939bc14a88b823ee357a52bded P 76ee766245ac40bfaa950ec259907c51 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260621 05:10:39.873103 8720 raft_consensus.cc:2258] T 561c27939bc14a88b823ee357a52bded P 76ee766245ac40bfaa950ec259907c51 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 05:10:39.874022 8721 ts_tablet_manager.cc:1905] T 561c27939bc14a88b823ee357a52bded P dcb5df66fb6a47cba61574c47f874330: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 05:10:39.876191 8721 ts_tablet_manager.cc:1918] T 561c27939bc14a88b823ee357a52bded P dcb5df66fb6a47cba61574c47f874330: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260621 05:10:39.876277 8721 log.cc:1196] T 561c27939bc14a88b823ee357a52bded P dcb5df66fb6a47cba61574c47f874330: Deleting WAL directory at /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782018616648578-3598-0/minicluster-data/ts-1/wal/wals/561c27939bc14a88b823ee357a52bded
I20260621 05:10:39.876544 8721 ts_tablet_manager.cc:1939] T 561c27939bc14a88b823ee357a52bded P dcb5df66fb6a47cba61574c47f874330: Deleting consensus metadata
I20260621 05:10:39.877044 8720 ts_tablet_manager.cc:1905] T 561c27939bc14a88b823ee357a52bded P 76ee766245ac40bfaa950ec259907c51: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 05:10:39.877204 8217 catalog_manager.cc:4841] TS dcb5df66fb6a47cba61574c47f874330 (127.3.131.130:35439): tablet 561c27939bc14a88b823ee357a52bded (table test-table [id=dfafba98da6e4082b91f606c57db05af]) successfully deleted
I20260621 05:10:39.879081 8720 ts_tablet_manager.cc:1918] T 561c27939bc14a88b823ee357a52bded P 76ee766245ac40bfaa950ec259907c51: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260621 05:10:39.879208 8720 log.cc:1196] T 561c27939bc14a88b823ee357a52bded P 76ee766245ac40bfaa950ec259907c51: Deleting WAL directory at /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782018616648578-3598-0/minicluster-data/ts-2/wal/wals/561c27939bc14a88b823ee357a52bded
I20260621 05:10:39.879472 8720 ts_tablet_manager.cc:1939] T 561c27939bc14a88b823ee357a52bded P 76ee766245ac40bfaa950ec259907c51: Deleting consensus metadata
I20260621 05:10:39.880023 8218 catalog_manager.cc:4841] TS 76ee766245ac40bfaa950ec259907c51 (127.3.131.131:35235): tablet 561c27939bc14a88b823ee357a52bded (table test-table [id=dfafba98da6e4082b91f606c57db05af]) successfully deleted
W20260621 05:10:39.880798 8220 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.3.131.129:43157 (error 108)
I20260621 05:10:39.881083 3598 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWZOx2_/build/release/bin/kudu with pid 8424
I20260621 05:10:39.881697 8693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:10:39.881256 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.131.129:43157 (local address 127.0.0.1:36460)
0621 05:10:39.881608 (+ 352us) negotiation.cc:107] Waiting for socket to connect
0621 05:10:39.881653 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.131.129:43157: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":330}
W20260621 05:10:39.881945 8220 catalog_manager.cc:4584] TS 3e5d2bacdfde4aba95becffd65fb03fb (127.3.131.129:43157): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 561c27939bc14a88b823ee357a52bded: Network error: Client connection negotiation failed: client connection to 127.3.131.129:43157: connect: Connection refused (error 111)
I20260621 05:10:39.886196 3598 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWZOx2_/build/release/bin/kudu with pid 8555
I20260621 05:10:39.891027 3598 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWZOx2_/build/release/bin/kudu with pid 8202
2026-06-21T05:10:39Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (824 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260621 05:10:39.911219 3598 mini_oidc.cc:152] Starting JWKS server
I20260621 05:10:39.912309 3598 webserver.cc:460] Webserver started at https://127.0.0.1:43603/ using document root <none> and password file <none>
I20260621 05:10:39.912437 3598 mini_oidc.cc:189] Starting OIDC Discovery server
I20260621 05:10:39.912634 3598 webserver.cc:460] Webserver started at http://127.0.0.1:33115/ using document root <none> and password file <none>
2026-06-21T05:10:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-21T05:10:39Z Disabled control of system clock
I20260621 05:10:39.921317 3598 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWZOx2_/build/release/bin/kudu
/tmp/dist-test-taskWZOx2_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1782018616648578-3598-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1782018616648578-3598-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1782018616648578-3598-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1782018616648578-3598-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.131.190:43383
--webserver_interface=127.3.131.190
--webserver_port=0
--builtin_ntp_servers=127.3.131.148:38669
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.131.190:43383
--enable_jwt_token_auth=true
--jwks_url=https://localhost:43603/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1782018616648578-3598-0/testchainca.cert with env {}
W20260621 05:10:40.007860 8729 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 05:10:40.008103 8729 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 05:10:40.008128 8729 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 05:10:40.009366 8729 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 05:10:40.009418 8729 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 05:10:40.009430 8729 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 05:10:40.009443 8729 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 05:10:40.009455 8729 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260621 05:10:40.009469 8729 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260621 05:10:40.009487 8729 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:43603/jwks.json
I20260621 05:10:40.011134 8729 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.131.148:38669
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1782018616648578-3598-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1782018616648578-3598-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.131.190:43383
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.131.190:43383
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:43603/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1782018616648578-3598-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.131.190
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1782018616648578-3598-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8729
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1782018616648578-3598-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 21 Jun 2026 04:16:16 UTC on ccycloud.quasar-xycaha.root.comops.site
build id 4457881
I20260621 05:10:40.011391 8729 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 05:10:40.011624 8729 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260621 05:10:40.017590 8729 server_base.cc:1034] running on GCE node
W20260621 05:10:40.017590 8737 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:10:40.017792 8735 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:10:40.018026 8734 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:10:40.018452 8729 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 05:10:40.018795 8729 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 05:10:40.020006 8729 hybrid_clock.cc:648] HybridClock initialized: now 1782018640019965 us; error 55 us; skew 500 ppm
I20260621 05:10:40.021759 8729 webserver.cc:460] Webserver started at http://127.3.131.190:39877/ using document root <none> and password file <none>
I20260621 05:10:40.022055 8729 fs_manager.cc:359] Metadata directory not provided
I20260621 05:10:40.022135 8729 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 05:10:40.022312 8729 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 05:10:40.023705 8729 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1782018616648578-3598-0/minicluster-data/master-0/data/instance:
uuid: "5b8675951752491b853d7eec18e8a5d5"
format_stamp: "Formatted at 2026-06-21 05:10:40 on dist-test-slave-vrq1"
server_key: "a2fe5da63980e61e76cc06bddfdfd60b"
server_key_iv: "a012df0c62a5dc08426d9e93a0661a6e"
server_key_version: "encryptionkey@0"
I20260621 05:10:40.024132 8729 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1782018616648578-3598-0/minicluster-data/master-0/wal/instance:
uuid: "5b8675951752491b853d7eec18e8a5d5"
format_stamp: "Formatted at 2026-06-21 05:10:40 on dist-test-slave-vrq1"
server_key: "a2fe5da63980e61e76cc06bddfdfd60b"
server_key_iv: "a012df0c62a5dc08426d9e93a0661a6e"
server_key_version: "encryptionkey@0"
I20260621 05:10:40.025667 8729 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260621 05:10:40.026671 8743 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 05:10:40.026934 8729 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260621 05:10:40.027122 8729 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1782018616648578-3598-0/minicluster-data/master-0/data,/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1782018616648578-3598-0/minicluster-data/master-0/wal
uuid: "5b8675951752491b853d7eec18e8a5d5"
format_stamp: "Formatted at 2026-06-21 05:10:40 on dist-test-slave-vrq1"
server_key: "a2fe5da63980e61e76cc06bddfdfd60b"
server_key_iv: "a012df0c62a5dc08426d9e93a0661a6e"
server_key_version: "encryptionkey@0"
I20260621 05:10:40.027243 8729 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1782018616648578-3598-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1782018616648578-3598-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1782018616648578-3598-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 05:10:40.040892 8729 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 05:10:40.041267 8729 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 05:10:40.041437 8729 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 05:10:40.045506 8729 rpc_server.cc:307] RPC server started. Bound to: 127.3.131.190:43383
I20260621 05:10:40.045539 8795 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.131.190:43383 every 8 connection(s)
I20260621 05:10:40.045895 8729 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1782018616648578-3598-0/minicluster-data/master-0/data/info.pb
I20260621 05:10:40.046356 3598 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWZOx2_/build/release/bin/kudu as pid 8729
I20260621 05:10:40.046470 3598 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1782018616648578-3598-0/minicluster-data/master-0/wal/instance
I20260621 05:10:40.046715 3598 external_mini_cluster.cc:1428] Setting key 88d4778c13aacc345ce62c97f5f5fc21
I20260621 05:10:40.047066 8796 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 05:10:40.050541 8796 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5b8675951752491b853d7eec18e8a5d5: Bootstrap starting.
I20260621 05:10:40.051306 8796 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5b8675951752491b853d7eec18e8a5d5: Neither blocks nor log segments found. Creating new log.
I20260621 05:10:40.051604 8796 log.cc:824] T 00000000000000000000000000000000 P 5b8675951752491b853d7eec18e8a5d5: Log is configured to *not* fsync() on all Append() calls
I20260621 05:10:40.052524 8796 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5b8675951752491b853d7eec18e8a5d5: No bootstrap required, opened a new log
I20260621 05:10:40.053033 8799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:10:40.047311 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36148 (local address 127.3.131.190:43383)
0621 05:10:40.047565 (+ 254us) server_negotiation.cc:206] Beginning negotiation
0621 05:10:40.047572 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0621 05:10:40.047608 (+ 36us) server_negotiation.cc:407] Connection header received
0621 05:10:40.048286 (+ 678us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:10:40.048295 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:10:40.048364 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:10:40.048549 (+ 185us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:10:40.048843 (+ 294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:10:40.049781 (+ 938us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:10:40.050879 (+ 1098us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:10:40.051442 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:10:40.051593 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:10:40.052376 (+ 783us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:10:40.052380 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:10:40.052381 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 05:10:40.052543 (+ 162us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:10:40.052604 (+ 61us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 05:10:40.052622 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:10:40.052700 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0621 05:10:40.052780 (+ 80us) server_negotiation.cc:299] Negotiation successful
0621 05:10:40.052861 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":187,"thread_start_us":102,"threads_started":1}
I20260621 05:10:40.054162 3598 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260621 05:10:40.055275 8796 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5b8675951752491b853d7eec18e8a5d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b8675951752491b853d7eec18e8a5d5" member_type: VOTER last_known_addr { host: "127.3.131.190" port: 43383 } }
I20260621 05:10:40.055451 8796 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5b8675951752491b853d7eec18e8a5d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 05:10:40.055496 8796 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5b8675951752491b853d7eec18e8a5d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b8675951752491b853d7eec18e8a5d5, State: Initialized, Role: FOLLOWER
I20260621 05:10:40.055627 8796 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5b8675951752491b853d7eec18e8a5d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b8675951752491b853d7eec18e8a5d5" member_type: VOTER last_known_addr { host: "127.3.131.190" port: 43383 } }
I20260621 05:10:40.055745 8796 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5b8675951752491b853d7eec18e8a5d5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260621 05:10:40.055800 8796 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5b8675951752491b853d7eec18e8a5d5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260621 05:10:40.055848 8796 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5b8675951752491b853d7eec18e8a5d5 [term 0 FOLLOWER]: Advancing to term 1
I20260621 05:10:40.056454 8796 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5b8675951752491b853d7eec18e8a5d5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b8675951752491b853d7eec18e8a5d5" member_type: VOTER last_known_addr { host: "127.3.131.190" port: 43383 } }
I20260621 05:10:40.056576 8796 leader_election.cc:302] T 00000000000000000000000000000000 P 5b8675951752491b853d7eec18e8a5d5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5b8675951752491b853d7eec18e8a5d5; no voters:
I20260621 05:10:40.056774 8796 leader_election.cc:288] T 00000000000000000000000000000000 P 5b8675951752491b853d7eec18e8a5d5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260621 05:10:40.056900 8801 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5b8675951752491b853d7eec18e8a5d5 [term 1 FOLLOWER]: Leader election won for term 1
I20260621 05:10:40.057057 8796 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5b8675951752491b853d7eec18e8a5d5 [sys.catalog]: configured and running, proceeding with master startup.
I20260621 05:10:40.057080 8801 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5b8675951752491b853d7eec18e8a5d5 [term 1 LEADER]: Becoming Leader. State: Replica: 5b8675951752491b853d7eec18e8a5d5, State: Running, Role: LEADER
I20260621 05:10:40.057168 8801 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5b8675951752491b853d7eec18e8a5d5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b8675951752491b853d7eec18e8a5d5" member_type: VOTER last_known_addr { host: "127.3.131.190" port: 43383 } }
I20260621 05:10:40.057578 8803 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5b8675951752491b853d7eec18e8a5d5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5b8675951752491b853d7eec18e8a5d5. Latest consensus state: current_term: 1 leader_uuid: "5b8675951752491b853d7eec18e8a5d5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b8675951752491b853d7eec18e8a5d5" member_type: VOTER last_known_addr { host: "127.3.131.190" port: 43383 } } }
I20260621 05:10:40.057683 8803 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5b8675951752491b853d7eec18e8a5d5 [sys.catalog]: This master's current role is: LEADER
I20260621 05:10:40.057852 8802 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5b8675951752491b853d7eec18e8a5d5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5b8675951752491b853d7eec18e8a5d5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b8675951752491b853d7eec18e8a5d5" member_type: VOTER last_known_addr { host: "127.3.131.190" port: 43383 } } }
I20260621 05:10:40.057898 8810 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260621 05:10:40.057927 8802 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5b8675951752491b853d7eec18e8a5d5 [sys.catalog]: This master's current role is: LEADER
I20260621 05:10:40.058423 8810 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260621 05:10:40.060060 8810 catalog_manager.cc:1347] Generated new cluster ID: f50d864ceb6b497e8cf564a193a99a0f
I20260621 05:10:40.060124 8810 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260621 05:10:40.070894 8810 catalog_manager.cc:1370] Generated new certificate authority record
I20260621 05:10:40.071523 8810 catalog_manager.cc:1504] Loading token signing keys...
I20260621 05:10:40.075047 8810 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5b8675951752491b853d7eec18e8a5d5: Generated new TSK 0
I20260621 05:10:40.075273 8810 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260621 05:10:40.113588 8826 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260621 05:10:40.113734 8826 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260621 05:10:40.113986 8799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:10:40.091496 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36162 (local address 127.3.131.190:43383)
0621 05:10:40.091580 (+ 84us) server_negotiation.cc:206] Beginning negotiation
0621 05:10:40.091583 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0621 05:10:40.091746 (+ 163us) server_negotiation.cc:407] Connection header received
0621 05:10:40.091893 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:10:40.091899 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:10:40.091910 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:10:40.091977 (+ 67us) server_negotiation.cc:226] Negotiated authn=JWT
0621 05:10:40.092141 (+ 164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:10:40.092638 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:10:40.093453 (+ 815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:10:40.093644 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:10:40.093739 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:10:40.093989 (+ 250us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0621 05:10:40.113723 (+ 19734us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43603/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0621 05:10:40.113915 (+ 192us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:36162: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43603/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":44}
W20260621 05:10:40.114051 8799 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:36162: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43603/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260621 05:10:40.114377 8825 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 05:10:40.091423 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.131.190:43383 (local address 127.0.0.1:36162)
0621 05:10:40.091657 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0621 05:10:40.091667 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0621 05:10:40.091771 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:10:40.091986 (+ 215us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:10:40.091987 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:10:40.091989 (+ 2us) client_negotiation.cc:189] Negotiated authn=JWT
0621 05:10:40.092063 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:10:40.092065 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:10:40.092743 (+ 678us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:10:40.092744 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:10:40.093360 (+ 616us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:10:40.093362 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:10:40.093756 (+ 394us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:10:40.093757 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:10:40.093823 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:10:40.093849 (+ 26us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0621 05:10:40.113941 (+ 20092us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43603/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0621 05:10:40.114065 (+ 124us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.3.131.190:43383: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43603/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":176,"thread_start_us":110,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.3.131.190:43383: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43603/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260621 05:10:40.114913 3598 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWZOx2_/build/release/bin/kudu with pid 8729
2026-06-21T05:10:40Z chronyd exiting
I20260621 05:10:40.151743 3598 test_util.cc:175] -----------------------------------------------
I20260621 05:10:40.151829 3598 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1782018616648578-3598-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (241 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260621 05:10:40.152599 3598 mini_oidc.cc:152] Starting JWKS server
I20260621 05:10:40.153268 3598 webserver.cc:460] Webserver started at https://127.0.0.1:42139/ using document root <none> and password file <none>
I20260621 05:10:40.153309 3598 mini_oidc.cc:189] Starting OIDC Discovery server
I20260621 05:10:40.153457 3598 webserver.cc:460] Webserver started at http://127.0.0.1:33437/ using document root <none> and password file <none>
2026-06-21T05:10:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-21T05:10:40Z Disabled control of system clock
I20260621 05:10:40.161803 3598 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWZOx2_/build/release/bin/kudu
/tmp/dist-test-taskWZOx2_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1782018616648578-3598-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1782018616648578-3598-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1782018616648578-3598-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1782018616648578-3598-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.131.190:40517
--webserver_interface=127.3.131.190
--webserver_port=0
--builtin_ntp_servers=127.3.131.148:40305
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.131.190:40517
--enable_jwt_token_auth=true
--jwks_url=https://localhost:42139/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1782018616648578-3598-0/testchainca.cert with env {}
W20260621 05:10:40.249460 8832 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 05:10:40.249749 8832 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 05:10:40.249791 8832 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 05:10:40.251055 8832 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 05:10:40.251144 8832 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 05:10:40.251168 8832 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 05:10:40.251221 8832 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 05:10:40.251255 8832 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260621 05:10:40.251286 8832 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260621 05:10:40.251312 8832 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:42139/jwks.json
I20260621 05:10:40.253057 8832 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.131.148:40305
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1782018616648578-3598-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1782018616648578-3598-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.131.190:40517
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.131.190:40517
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:42139/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1782018616648578-3598-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.131.190
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1782018616648578-3598-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8832
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1782018616648578-3598-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 21 Jun 2026 04:16:16 UTC on ccycloud.quasar-xycaha.root.comops.site
build id 4457881
I20260621 05:10:40.253412 8832 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 05:10:40.253660 8832 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 05:10:40.258890 8840 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:10:40.259002 8838 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:10:40.258960 8837 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:10:40.259104 8832 server_base.cc:1034] running on GCE node
I20260621 05:10:40.259491 8832 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 05:10:40.259769 8832 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 05:10:40.260969 8832 hybrid_clock.cc:648] HybridClock initialized: now 1782018640260951 us; error 31 us; skew 500 ppm
I20260621 05:10:40.262430 8832 webserver.cc:460] Webserver started at http://127.3.131.190:45417/ using document root <none> and password file <none>
I20260621 05:10:40.262694 8832 fs_manager.cc:359] Metadata directory not provided
I20260621 05:10:40.262768 8832 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 05:10:40.262918 8832 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 05:10:40.264122 8832 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1782018616648578-3598-0/minicluster-data/master-0/data/instance:
uuid: "0f2214dd1c1b4b32a001192e2eb3bfd9"
format_stamp: "Formatted at 2026-06-21 05:10:40 on dist-test-slave-vrq1"
server_key: "d07e16fbe29472d569c3cd592c34aeed"
server_key_iv: "f3817df1ea800f26877ba1657715d4ee"
server_key_version: "encryptionkey@0"
I20260621 05:10:40.264510 8832 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1782018616648578-3598-0/minicluster-data/master-0/wal/instance:
uuid: "0f2214dd1c1b4b32a001192e2eb3bfd9"
format_stamp: "Formatted at 2026-06-21 05:10:40 on dist-test-slave-vrq1"
server_key: "d07e16fbe29472d569c3cd592c34aeed"
server_key_iv: "f3817df1ea800f26877ba1657715d4ee"
server_key_version: "encryptionkey@0"
I20260621 05:10:40.266187 8832 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260621 05:10:40.267366 8846 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 05:10:40.267614 8832 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260621 05:10:40.267732 8832 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1782018616648578-3598-0/minicluster-data/master-0/data,/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1782018616648578-3598-0/minicluster-data/master-0/wal
uuid: "0f2214dd1c1b4b32a001192e2eb3bfd9"
format_stamp: "Formatted at 2026-06-21 05:10:40 on dist-test-slave-vrq1"
server_key: "d07e16fbe29472d569c3cd592c34aeed"
server_key_iv: "f3817df1ea800f26877ba1657715d4ee"
server_key_version: "encryptionkey@0"
I20260621 05:10:40.267851 8832 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1782018616648578-3598-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1782018616648578-3598-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1782018616648578-3598-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 05:10:40.277858 8832 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 05:10:40.278251 8832 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 05:10:40.278419 8832 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 05:10:40.282449 8832 rpc_server.cc:307] RPC server started. Bound to: 127.3.131.190:40517
I20260621 05:10:40.282543 8898 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.131.190:40517 every 8 connection(s)
I20260621 05:10:40.282892 8832 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1782018616648578-3598-0/minicluster-data/master-0/data/info.pb
I20260621 05:10:40.284013 8899 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 05:10:40.286516 8899 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0f2214dd1c1b4b32a001192e2eb3bfd9: Bootstrap starting.
I20260621 05:10:40.286876 3598 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWZOx2_/build/release/bin/kudu as pid 8832
I20260621 05:10:40.287037 3598 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1782018616648578-3598-0/minicluster-data/master-0/wal/instance
I20260621 05:10:40.287281 3598 external_mini_cluster.cc:1428] Setting key fa543cd1c8be58ff43e9e773061e84c7
I20260621 05:10:40.287302 8899 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0f2214dd1c1b4b32a001192e2eb3bfd9: Neither blocks nor log segments found. Creating new log.
I20260621 05:10:40.287673 8899 log.cc:824] T 00000000000000000000000000000000 P 0f2214dd1c1b4b32a001192e2eb3bfd9: Log is configured to *not* fsync() on all Append() calls
I20260621 05:10:40.288551 8899 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0f2214dd1c1b4b32a001192e2eb3bfd9: No bootstrap required, opened a new log
I20260621 05:10:40.291118 8899 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0f2214dd1c1b4b32a001192e2eb3bfd9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0f2214dd1c1b4b32a001192e2eb3bfd9" member_type: VOTER last_known_addr { host: "127.3.131.190" port: 40517 } }
I20260621 05:10:40.291345 8899 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0f2214dd1c1b4b32a001192e2eb3bfd9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 05:10:40.291395 8899 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0f2214dd1c1b4b32a001192e2eb3bfd9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0f2214dd1c1b4b32a001192e2eb3bfd9, State: Initialized, Role: FOLLOWER
I20260621 05:10:40.291574 8899 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0f2214dd1c1b4b32a001192e2eb3bfd9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0f2214dd1c1b4b32a001192e2eb3bfd9" member_type: VOTER last_known_addr { host: "127.3.131.190" port: 40517 } }
I20260621 05:10:40.291705 8899 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0f2214dd1c1b4b32a001192e2eb3bfd9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260621 05:10:40.291754 8899 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0f2214dd1c1b4b32a001192e2eb3bfd9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260621 05:10:40.291811 8899 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0f2214dd1c1b4b32a001192e2eb3bfd9 [term 0 FOLLOWER]: Advancing to term 1
I20260621 05:10:40.292440 8899 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0f2214dd1c1b4b32a001192e2eb3bfd9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0f2214dd1c1b4b32a001192e2eb3bfd9" member_type: VOTER last_known_addr { host: "127.3.131.190" port: 40517 } }
I20260621 05:10:40.292562 8899 leader_election.cc:302] T 00000000000000000000000000000000 P 0f2214dd1c1b4b32a001192e2eb3bfd9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0f2214dd1c1b4b32a001192e2eb3bfd9; no voters:
I20260621 05:10:40.292737 8899 leader_election.cc:288] T 00000000000000000000000000000000 P 0f2214dd1c1b4b32a001192e2eb3bfd9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260621 05:10:40.292940 8905 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0f2214dd1c1b4b32a001192e2eb3bfd9 [term 1 FOLLOWER]: Leader election won for term 1
I20260621 05:10:40.293042 8899 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0f2214dd1c1b4b32a001192e2eb3bfd9 [sys.catalog]: configured and running, proceeding with master startup.
I20260621 05:10:40.293179 8905 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0f2214dd1c1b4b32a001192e2eb3bfd9 [term 1 LEADER]: Becoming Leader. State: Replica: 0f2214dd1c1b4b32a001192e2eb3bfd9, State: Running, Role: LEADER
I20260621 05:10:40.293310 8905 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0f2214dd1c1b4b32a001192e2eb3bfd9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0f2214dd1c1b4b32a001192e2eb3bfd9" member_type: VOTER last_known_addr { host: "127.3.131.190" port: 40517 } }
I20260621 05:10:40.293671 8905 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0f2214dd1c1b4b32a001192e2eb3bfd9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0f2214dd1c1b4b32a001192e2eb3bfd9. Latest consensus state: current_term: 1 leader_uuid: "0f2214dd1c1b4b32a001192e2eb3bfd9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0f2214dd1c1b4b32a001192e2eb3bfd9" member_type: VOTER last_known_addr { host: "127.3.131.190" port: 40517 } } }
I20260621 05:10:40.293830 8904 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0f2214dd1c1b4b32a001192e2eb3bfd9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0f2214dd1c1b4b32a001192e2eb3bfd9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0f2214dd1c1b4b32a001192e2eb3bfd9" member_type: VOTER last_known_addr { host: "127.3.131.190" port: 40517 } } }
I20260621 05:10:40.293752 8905 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0f2214dd1c1b4b32a001192e2eb3bfd9 [sys.catalog]: This master's current role is: LEADER
I20260621 05:10:40.293919 8904 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0f2214dd1c1b4b32a001192e2eb3bfd9 [sys.catalog]: This master's current role is: LEADER
I20260621 05:10:40.294463 8903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:10:40.287891 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58486 (local address 127.3.131.190:40517)
0621 05:10:40.288176 (+ 285us) server_negotiation.cc:206] Beginning negotiation
0621 05:10:40.288186 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0621 05:10:40.288219 (+ 33us) server_negotiation.cc:407] Connection header received
0621 05:10:40.288870 (+ 651us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:10:40.288880 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:10:40.288963 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:10:40.289135 (+ 172us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:10:40.289349 (+ 214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:10:40.290833 (+ 1484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:10:40.291896 (+ 1063us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:10:40.292340 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:10:40.292438 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:10:40.293281 (+ 843us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:10:40.293285 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:10:40.293285 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 05:10:40.293497 (+ 212us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:10:40.293553 (+ 56us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 05:10:40.293573 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:10:40.293641 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0621 05:10:40.294018 (+ 377us) server_negotiation.cc:299] Negotiation successful
0621 05:10:40.294123 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":195,"thread_start_us":120,"threads_started":1}
I20260621 05:10:40.294286 8914 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260621 05:10:40.295370 8914 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260621 05:10:40.296353 3598 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260621 05:10:40.303264 8914 catalog_manager.cc:1347] Generated new cluster ID: b8bf127da4144fbd98717b9dff6459bb
I20260621 05:10:40.303452 8914 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260621 05:10:40.313135 8914 catalog_manager.cc:1370] Generated new certificate authority record
I20260621 05:10:40.314069 8914 catalog_manager.cc:1504] Loading token signing keys...
I20260621 05:10:40.322615 8914 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0f2214dd1c1b4b32a001192e2eb3bfd9: Generated new TSK 0
I20260621 05:10:40.322876 8914 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260621 05:10:40.358443 8929 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260621 05:10:40.358542 8929 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
W20260621 05:10:40.359047 8928 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 05:10:40.335274 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.131.190:40517 (local address 127.0.0.1:58490)
0621 05:10:40.335520 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0621 05:10:40.335533 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0621 05:10:40.335604 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:10:40.335862 (+ 258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:10:40.335863 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:10:40.335865 (+ 2us) client_negotiation.cc:189] Negotiated authn=JWT
0621 05:10:40.335934 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:10:40.335937 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:10:40.337248 (+ 1311us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:10:40.337250 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:10:40.338238 (+ 988us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:10:40.338242 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:10:40.338675 (+ 433us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:10:40.338676 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:10:40.338740 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:10:40.338746 (+ 6us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0621 05:10:40.358809 (+ 20063us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42139/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0621 05:10:40.358935 (+ 126us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.3.131.190:40517: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42139/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":214,"thread_start_us":109,"threads_started":1}
I20260621 05:10:40.359062 8903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:10:40.335434 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58490 (local address 127.3.131.190:40517)
0621 05:10:40.335684 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0621 05:10:40.335686 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:10:40.335697 (+ 11us) server_negotiation.cc:407] Connection header received
0621 05:10:40.335752 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:10:40.335753 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:10:40.335764 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:10:40.335839 (+ 75us) server_negotiation.cc:226] Negotiated authn=JWT
0621 05:10:40.336098 (+ 259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:10:40.337013 (+ 915us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:10:40.338345 (+ 1332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:10:40.338586 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:10:40.338651 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:10:40.338850 (+ 199us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0621 05:10:40.358657 (+ 19807us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42139/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0621 05:10:40.358826 (+ 169us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:58490: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42139/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":80}
W20260621 05:10:40.359251 8903 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:58490: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42139/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
I20260621 05:10:40.359789 3598 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWZOx2_/build/release/bin/kudu with pid 8832
2026-06-21T05:10:40Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (245 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260621 05:10:40.398288 3598 mini_oidc.cc:152] Starting JWKS server
I20260621 05:10:40.399012 3598 webserver.cc:460] Webserver started at https://127.0.0.1:36701/ using document root <none> and password file <none>
I20260621 05:10:40.399084 3598 mini_oidc.cc:189] Starting OIDC Discovery server
I20260621 05:10:40.399319 3598 webserver.cc:460] Webserver started at http://127.0.0.1:37799/ using document root <none> and password file <none>
2026-06-21T05:10:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-21T05:10:40Z Disabled control of system clock
I20260621 05:10:40.409154 3598 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWZOx2_/build/release/bin/kudu
/tmp/dist-test-taskWZOx2_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1782018616648578-3598-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1782018616648578-3598-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1782018616648578-3598-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1782018616648578-3598-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.131.190:34439
--webserver_interface=127.3.131.190
--webserver_port=0
--builtin_ntp_servers=127.3.131.148:37725
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.131.190:34439
--enable_jwt_token_auth=true
--jwks_url=https://localhost:36701/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260621 05:10:40.498799 8935 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 05:10:40.499033 8935 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 05:10:40.499058 8935 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 05:10:40.500547 8935 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 05:10:40.500605 8935 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 05:10:40.500620 8935 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 05:10:40.500634 8935 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 05:10:40.500646 8935 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260621 05:10:40.500659 8935 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260621 05:10:40.500670 8935 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:36701/jwks.json
I20260621 05:10:40.502187 8935 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.131.148:37725
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1782018616648578-3598-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1782018616648578-3598-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.131.190:34439
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.131.190:34439
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:36701/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1782018616648578-3598-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.131.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8935
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1782018616648578-3598-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 21 Jun 2026 04:16:16 UTC on ccycloud.quasar-xycaha.root.comops.site
build id 4457881
I20260621 05:10:40.502470 8935 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 05:10:40.502789 8935 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 05:10:40.508268 8941 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:10:40.508279 8940 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:10:40.508262 8943 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:10:40.508526 8935 server_base.cc:1034] running on GCE node
I20260621 05:10:40.508810 8935 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 05:10:40.509130 8935 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 05:10:40.510363 8935 hybrid_clock.cc:648] HybridClock initialized: now 1782018640510285 us; error 93 us; skew 500 ppm
I20260621 05:10:40.511982 8935 webserver.cc:460] Webserver started at http://127.3.131.190:46511/ using document root <none> and password file <none>
I20260621 05:10:40.512295 8935 fs_manager.cc:359] Metadata directory not provided
I20260621 05:10:40.512388 8935 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 05:10:40.512550 8935 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 05:10:40.513710 8935 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1782018616648578-3598-0/minicluster-data/master-0/data/instance:
uuid: "1114238fdcc041dbbf70015d1371ac5a"
format_stamp: "Formatted at 2026-06-21 05:10:40 on dist-test-slave-vrq1"
server_key: "eb5bd67dbea726ce03cbab231c3af803"
server_key_iv: "8b9e1a5d674f40c117a48f6895980059"
server_key_version: "encryptionkey@0"
I20260621 05:10:40.514087 8935 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1782018616648578-3598-0/minicluster-data/master-0/wal/instance:
uuid: "1114238fdcc041dbbf70015d1371ac5a"
format_stamp: "Formatted at 2026-06-21 05:10:40 on dist-test-slave-vrq1"
server_key: "eb5bd67dbea726ce03cbab231c3af803"
server_key_iv: "8b9e1a5d674f40c117a48f6895980059"
server_key_version: "encryptionkey@0"
I20260621 05:10:40.515559 8935 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.001s
I20260621 05:10:40.516637 8949 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 05:10:40.517119 8935 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260621 05:10:40.517271 8935 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1782018616648578-3598-0/minicluster-data/master-0/data,/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1782018616648578-3598-0/minicluster-data/master-0/wal
uuid: "1114238fdcc041dbbf70015d1371ac5a"
format_stamp: "Formatted at 2026-06-21 05:10:40 on dist-test-slave-vrq1"
server_key: "eb5bd67dbea726ce03cbab231c3af803"
server_key_iv: "8b9e1a5d674f40c117a48f6895980059"
server_key_version: "encryptionkey@0"
I20260621 05:10:40.517370 8935 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1782018616648578-3598-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1782018616648578-3598-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1782018616648578-3598-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 05:10:40.527802 8935 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 05:10:40.528218 8935 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 05:10:40.528411 8935 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 05:10:40.533355 8935 rpc_server.cc:307] RPC server started. Bound to: 127.3.131.190:34439
I20260621 05:10:40.533388 9001 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.131.190:34439 every 8 connection(s)
I20260621 05:10:40.533830 8935 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1782018616648578-3598-0/minicluster-data/master-0/data/info.pb
I20260621 05:10:40.534507 9002 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 05:10:40.534894 3598 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWZOx2_/build/release/bin/kudu as pid 8935
I20260621 05:10:40.534981 3598 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1782018616648578-3598-0/minicluster-data/master-0/wal/instance
I20260621 05:10:40.535193 3598 external_mini_cluster.cc:1428] Setting key c171fc57948d0ce429e181093610d229
I20260621 05:10:40.539162 9002 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1114238fdcc041dbbf70015d1371ac5a: Bootstrap starting.
I20260621 05:10:40.539935 9002 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1114238fdcc041dbbf70015d1371ac5a: Neither blocks nor log segments found. Creating new log.
I20260621 05:10:40.540266 9002 log.cc:824] T 00000000000000000000000000000000 P 1114238fdcc041dbbf70015d1371ac5a: Log is configured to *not* fsync() on all Append() calls
I20260621 05:10:40.541148 9002 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1114238fdcc041dbbf70015d1371ac5a: No bootstrap required, opened a new log
I20260621 05:10:40.542281 9005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:10:40.535865 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58894 (local address 127.3.131.190:34439)
0621 05:10:40.536116 (+ 251us) server_negotiation.cc:206] Beginning negotiation
0621 05:10:40.536128 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0621 05:10:40.536166 (+ 38us) server_negotiation.cc:407] Connection header received
0621 05:10:40.537004 (+ 838us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:10:40.537015 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:10:40.537115 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:10:40.537303 (+ 188us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:10:40.537555 (+ 252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:10:40.539066 (+ 1511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:10:40.540113 (+ 1047us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:10:40.540613 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:10:40.540771 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:10:40.541604 (+ 833us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:10:40.541608 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:10:40.541608 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 05:10:40.541776 (+ 168us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:10:40.541811 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 05:10:40.541823 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:10:40.541897 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0621 05:10:40.542013 (+ 116us) server_negotiation.cc:299] Negotiation successful
0621 05:10:40.542108 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":160,"thread_start_us":101,"threads_started":1}
I20260621 05:10:40.543772 3598 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260621 05:10:40.543762 9002 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1114238fdcc041dbbf70015d1371ac5a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1114238fdcc041dbbf70015d1371ac5a" member_type: VOTER last_known_addr { host: "127.3.131.190" port: 34439 } }
I20260621 05:10:40.543960 9002 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1114238fdcc041dbbf70015d1371ac5a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 05:10:40.544009 9002 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1114238fdcc041dbbf70015d1371ac5a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1114238fdcc041dbbf70015d1371ac5a, State: Initialized, Role: FOLLOWER
I20260621 05:10:40.544152 9002 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1114238fdcc041dbbf70015d1371ac5a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1114238fdcc041dbbf70015d1371ac5a" member_type: VOTER last_known_addr { host: "127.3.131.190" port: 34439 } }
I20260621 05:10:40.544243 9002 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1114238fdcc041dbbf70015d1371ac5a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260621 05:10:40.544291 9002 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1114238fdcc041dbbf70015d1371ac5a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260621 05:10:40.544344 9002 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1114238fdcc041dbbf70015d1371ac5a [term 0 FOLLOWER]: Advancing to term 1
I20260621 05:10:40.545171 9002 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1114238fdcc041dbbf70015d1371ac5a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1114238fdcc041dbbf70015d1371ac5a" member_type: VOTER last_known_addr { host: "127.3.131.190" port: 34439 } }
I20260621 05:10:40.545301 9002 leader_election.cc:302] T 00000000000000000000000000000000 P 1114238fdcc041dbbf70015d1371ac5a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1114238fdcc041dbbf70015d1371ac5a; no voters:
I20260621 05:10:40.545569 9002 leader_election.cc:288] T 00000000000000000000000000000000 P 1114238fdcc041dbbf70015d1371ac5a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260621 05:10:40.545749 9007 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1114238fdcc041dbbf70015d1371ac5a [term 1 FOLLOWER]: Leader election won for term 1
I20260621 05:10:40.545871 9002 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1114238fdcc041dbbf70015d1371ac5a [sys.catalog]: configured and running, proceeding with master startup.
I20260621 05:10:40.545938 9007 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1114238fdcc041dbbf70015d1371ac5a [term 1 LEADER]: Becoming Leader. State: Replica: 1114238fdcc041dbbf70015d1371ac5a, State: Running, Role: LEADER
I20260621 05:10:40.546150 9007 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1114238fdcc041dbbf70015d1371ac5a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1114238fdcc041dbbf70015d1371ac5a" member_type: VOTER last_known_addr { host: "127.3.131.190" port: 34439 } }
I20260621 05:10:40.546605 9008 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1114238fdcc041dbbf70015d1371ac5a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1114238fdcc041dbbf70015d1371ac5a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1114238fdcc041dbbf70015d1371ac5a" member_type: VOTER last_known_addr { host: "127.3.131.190" port: 34439 } } }
I20260621 05:10:40.546725 9008 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1114238fdcc041dbbf70015d1371ac5a [sys.catalog]: This master's current role is: LEADER
I20260621 05:10:40.546916 9009 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1114238fdcc041dbbf70015d1371ac5a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1114238fdcc041dbbf70015d1371ac5a. Latest consensus state: current_term: 1 leader_uuid: "1114238fdcc041dbbf70015d1371ac5a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1114238fdcc041dbbf70015d1371ac5a" member_type: VOTER last_known_addr { host: "127.3.131.190" port: 34439 } } }
I20260621 05:10:40.547000 9009 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1114238fdcc041dbbf70015d1371ac5a [sys.catalog]: This master's current role is: LEADER
I20260621 05:10:40.546978 9016 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260621 05:10:40.547865 9016 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260621 05:10:40.549552 9016 catalog_manager.cc:1347] Generated new cluster ID: c0bd766549f7446bbb56f86a113b2e29
I20260621 05:10:40.549615 9016 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260621 05:10:40.574631 9016 catalog_manager.cc:1370] Generated new certificate authority record
I20260621 05:10:40.575232 9016 catalog_manager.cc:1504] Loading token signing keys...
I20260621 05:10:40.582105 9016 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1114238fdcc041dbbf70015d1371ac5a: Generated new TSK 0
I20260621 05:10:40.582362 9016 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260621 05:10:40.602291 9005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:10:40.597421 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58908 (local address 127.3.131.190:34439)
0621 05:10:40.597469 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0621 05:10:40.597471 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:10:40.597615 (+ 144us) server_negotiation.cc:407] Connection header received
0621 05:10:40.597744 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:10:40.597746 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:10:40.597762 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:10:40.597842 (+ 80us) server_negotiation.cc:226] Negotiated authn=JWT
0621 05:10:40.598102 (+ 260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:10:40.598784 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:10:40.600505 (+ 1721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:10:40.600727 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:10:40.600793 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:10:40.601037 (+ 244us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0621 05:10:40.602067 (+ 1030us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36701/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0621 05:10:40.602212 (+ 145us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:58908: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36701/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":24}
W20260621 05:10:40.602366 9005 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:58908: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36701/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260621 05:10:40.602830 9031 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 05:10:40.597181 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.131.190:34439 (local address 127.0.0.1:58908)
0621 05:10:40.597540 (+ 359us) negotiation.cc:107] Waiting for socket to connect
0621 05:10:40.597552 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0621 05:10:40.597633 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:10:40.597903 (+ 270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:10:40.597905 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:10:40.597907 (+ 2us) client_negotiation.cc:189] Negotiated authn=JWT
0621 05:10:40.597973 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:10:40.597986 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:10:40.598913 (+ 927us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:10:40.598914 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:10:40.600213 (+ 1299us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:10:40.600218 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:10:40.600830 (+ 612us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:10:40.600831 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:10:40.600905 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:10:40.600912 (+ 7us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0621 05:10:40.602182 (+ 1270us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36701/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0621 05:10:40.602303 (+ 121us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.3.131.190:34439: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36701/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":331,"thread_start_us":106,"threads_started":1}
I20260621 05:10:40.603127 3598 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWZOx2_/build/release/bin/kudu with pid 8935
E20260621 05:10:40.613188 9032 webserver.cc:533] Webserver: sslize failed:
E20260621 05:10:40.613306 9032 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
2026-06-21T05:10:40Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (243 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-06-21T05:10:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-21T05:10:40Z Disabled control of system clock
I20260621 05:10:40.650583 3598 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWZOx2_/build/release/bin/kudu
/tmp/dist-test-taskWZOx2_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1782018616648578-3598-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1782018616648578-3598-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1782018616648578-3598-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1782018616648578-3598-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.131.190:41839
--webserver_interface=127.3.131.190
--webserver_port=0
--builtin_ntp_servers=127.3.131.148:36491
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.131.190:41839
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1782018616648578-3598-0/testchainca.cert with env {}
W20260621 05:10:40.738718 9036 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 05:10:40.738943 9036 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 05:10:40.738968 9036 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 05:10:40.740293 9036 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 05:10:40.740355 9036 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 05:10:40.740368 9036 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 05:10:40.740381 9036 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 05:10:40.740394 9036 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260621 05:10:40.740407 9036 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260621 05:10:40.740418 9036 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260621 05:10:40.742106 9036 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.131.148:36491
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1782018616648578-3598-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1782018616648578-3598-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.131.190:41839
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.131.190:41839
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1782018616648578-3598-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.131.190
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1782018616648578-3598-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9036
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1782018616648578-3598-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 21 Jun 2026 04:16:16 UTC on ccycloud.quasar-xycaha.root.comops.site
build id 4457881
I20260621 05:10:40.742403 9036 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 05:10:40.742810 9036 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 05:10:40.748927 9042 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:10:40.749006 9044 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:10:40.748927 9041 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:10:40.749068 9036 server_base.cc:1034] running on GCE node
I20260621 05:10:40.749593 9036 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 05:10:40.750103 9036 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 05:10:40.751307 9036 hybrid_clock.cc:648] HybridClock initialized: now 1782018640751276 us; error 48 us; skew 500 ppm
I20260621 05:10:40.752893 9036 webserver.cc:460] Webserver started at http://127.3.131.190:45967/ using document root <none> and password file <none>
I20260621 05:10:40.753194 9036 fs_manager.cc:359] Metadata directory not provided
I20260621 05:10:40.753279 9036 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 05:10:40.753448 9036 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 05:10:40.754675 9036 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1782018616648578-3598-0/minicluster-data/master-0/data/instance:
uuid: "bf799680f38f4eb7a7e763f4b2e6a046"
format_stamp: "Formatted at 2026-06-21 05:10:40 on dist-test-slave-vrq1"
server_key: "4f11fd08b1e5969000088037508dddce"
server_key_iv: "58c9b3b08cdc50cb1da92b24e3730890"
server_key_version: "encryptionkey@0"
I20260621 05:10:40.755103 9036 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1782018616648578-3598-0/minicluster-data/master-0/wal/instance:
uuid: "bf799680f38f4eb7a7e763f4b2e6a046"
format_stamp: "Formatted at 2026-06-21 05:10:40 on dist-test-slave-vrq1"
server_key: "4f11fd08b1e5969000088037508dddce"
server_key_iv: "58c9b3b08cdc50cb1da92b24e3730890"
server_key_version: "encryptionkey@0"
I20260621 05:10:40.756809 9036 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260621 05:10:40.757885 9050 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 05:10:40.758101 9036 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260621 05:10:40.758246 9036 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1782018616648578-3598-0/minicluster-data/master-0/data,/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1782018616648578-3598-0/minicluster-data/master-0/wal
uuid: "bf799680f38f4eb7a7e763f4b2e6a046"
format_stamp: "Formatted at 2026-06-21 05:10:40 on dist-test-slave-vrq1"
server_key: "4f11fd08b1e5969000088037508dddce"
server_key_iv: "58c9b3b08cdc50cb1da92b24e3730890"
server_key_version: "encryptionkey@0"
I20260621 05:10:40.758340 9036 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1782018616648578-3598-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1782018616648578-3598-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1782018616648578-3598-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 05:10:40.772475 9036 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 05:10:40.772833 9036 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 05:10:40.772993 9036 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 05:10:40.777565 9036 rpc_server.cc:307] RPC server started. Bound to: 127.3.131.190:41839
I20260621 05:10:40.777602 9102 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.131.190:41839 every 8 connection(s)
I20260621 05:10:40.778007 9036 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1782018616648578-3598-0/minicluster-data/master-0/data/info.pb
I20260621 05:10:40.779165 9103 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 05:10:40.781814 9103 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bf799680f38f4eb7a7e763f4b2e6a046: Bootstrap starting.
I20260621 05:10:40.782399 9103 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bf799680f38f4eb7a7e763f4b2e6a046: Neither blocks nor log segments found. Creating new log.
I20260621 05:10:40.782645 9103 log.cc:824] T 00000000000000000000000000000000 P bf799680f38f4eb7a7e763f4b2e6a046: Log is configured to *not* fsync() on all Append() calls
I20260621 05:10:40.783445 9103 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bf799680f38f4eb7a7e763f4b2e6a046: No bootstrap required, opened a new log
I20260621 05:10:40.785297 9103 raft_consensus.cc:348] T 00000000000000000000000000000000 P bf799680f38f4eb7a7e763f4b2e6a046 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf799680f38f4eb7a7e763f4b2e6a046" member_type: VOTER last_known_addr { host: "127.3.131.190" port: 41839 } }
I20260621 05:10:40.785465 9103 raft_consensus.cc:374] T 00000000000000000000000000000000 P bf799680f38f4eb7a7e763f4b2e6a046 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 05:10:40.785491 9103 raft_consensus.cc:729] T 00000000000000000000000000000000 P bf799680f38f4eb7a7e763f4b2e6a046 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bf799680f38f4eb7a7e763f4b2e6a046, State: Initialized, Role: FOLLOWER
I20260621 05:10:40.785667 9103 consensus_queue.cc:260] T 00000000000000000000000000000000 P bf799680f38f4eb7a7e763f4b2e6a046 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf799680f38f4eb7a7e763f4b2e6a046" member_type: VOTER last_known_addr { host: "127.3.131.190" port: 41839 } }
I20260621 05:10:40.785765 9103 raft_consensus.cc:388] T 00000000000000000000000000000000 P bf799680f38f4eb7a7e763f4b2e6a046 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260621 05:10:40.785790 9103 raft_consensus.cc:482] T 00000000000000000000000000000000 P bf799680f38f4eb7a7e763f4b2e6a046 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260621 05:10:40.785866 9103 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bf799680f38f4eb7a7e763f4b2e6a046 [term 0 FOLLOWER]: Advancing to term 1
I20260621 05:10:40.785910 3598 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWZOx2_/build/release/bin/kudu as pid 9036
I20260621 05:10:40.786001 3598 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1782018616648578-3598-0/minicluster-data/master-0/wal/instance
I20260621 05:10:40.786185 3598 external_mini_cluster.cc:1428] Setting key 653bd7229bcfbcba2a22aa1d7aa7f7e4
I20260621 05:10:40.786751 9103 raft_consensus.cc:504] T 00000000000000000000000000000000 P bf799680f38f4eb7a7e763f4b2e6a046 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf799680f38f4eb7a7e763f4b2e6a046" member_type: VOTER last_known_addr { host: "127.3.131.190" port: 41839 } }
I20260621 05:10:40.786876 9103 leader_election.cc:302] T 00000000000000000000000000000000 P bf799680f38f4eb7a7e763f4b2e6a046 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bf799680f38f4eb7a7e763f4b2e6a046; no voters:
I20260621 05:10:40.787091 9103 leader_election.cc:288] T 00000000000000000000000000000000 P bf799680f38f4eb7a7e763f4b2e6a046 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260621 05:10:40.787115 9108 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bf799680f38f4eb7a7e763f4b2e6a046 [term 1 FOLLOWER]: Leader election won for term 1
I20260621 05:10:40.787249 9108 raft_consensus.cc:686] T 00000000000000000000000000000000 P bf799680f38f4eb7a7e763f4b2e6a046 [term 1 LEADER]: Becoming Leader. State: Replica: bf799680f38f4eb7a7e763f4b2e6a046, State: Running, Role: LEADER
I20260621 05:10:40.787339 9108 consensus_queue.cc:237] T 00000000000000000000000000000000 P bf799680f38f4eb7a7e763f4b2e6a046 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf799680f38f4eb7a7e763f4b2e6a046" member_type: VOTER last_known_addr { host: "127.3.131.190" port: 41839 } }
I20260621 05:10:40.787406 9103 sys_catalog.cc:564] T 00000000000000000000000000000000 P bf799680f38f4eb7a7e763f4b2e6a046 [sys.catalog]: configured and running, proceeding with master startup.
I20260621 05:10:40.787715 9108 sys_catalog.cc:455] T 00000000000000000000000000000000 P bf799680f38f4eb7a7e763f4b2e6a046 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bf799680f38f4eb7a7e763f4b2e6a046" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf799680f38f4eb7a7e763f4b2e6a046" member_type: VOTER last_known_addr { host: "127.3.131.190" port: 41839 } } }
I20260621 05:10:40.787791 9108 sys_catalog.cc:458] T 00000000000000000000000000000000 P bf799680f38f4eb7a7e763f4b2e6a046 [sys.catalog]: This master's current role is: LEADER
I20260621 05:10:40.787768 9109 sys_catalog.cc:455] T 00000000000000000000000000000000 P bf799680f38f4eb7a7e763f4b2e6a046 [sys.catalog]: SysCatalogTable state changed. Reason: New leader bf799680f38f4eb7a7e763f4b2e6a046. Latest consensus state: current_term: 1 leader_uuid: "bf799680f38f4eb7a7e763f4b2e6a046" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf799680f38f4eb7a7e763f4b2e6a046" member_type: VOTER last_known_addr { host: "127.3.131.190" port: 41839 } } }
I20260621 05:10:40.787925 9109 sys_catalog.cc:458] T 00000000000000000000000000000000 P bf799680f38f4eb7a7e763f4b2e6a046 [sys.catalog]: This master's current role is: LEADER
I20260621 05:10:40.788563 9117 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260621 05:10:40.789101 9117 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260621 05:10:40.790711 9117 catalog_manager.cc:1347] Generated new cluster ID: 46e5b0b080f9437ca42a807387785ba4
I20260621 05:10:40.790774 9117 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260621 05:10:40.793164 9107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:10:40.786774 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52040 (local address 127.3.131.190:41839)
0621 05:10:40.787093 (+ 319us) server_negotiation.cc:206] Beginning negotiation
0621 05:10:40.787102 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0621 05:10:40.787162 (+ 60us) server_negotiation.cc:407] Connection header received
0621 05:10:40.787337 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:10:40.787344 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:10:40.787445 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:10:40.787608 (+ 163us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:10:40.787946 (+ 338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:10:40.789459 (+ 1513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:10:40.790742 (+ 1283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:10:40.791382 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:10:40.791505 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:10:40.792265 (+ 760us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:10:40.792269 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:10:40.792269 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 05:10:40.792488 (+ 219us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:10:40.792534 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 05:10:40.792560 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:10:40.792639 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0621 05:10:40.792718 (+ 79us) server_negotiation.cc:299] Negotiation successful
0621 05:10:40.792914 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":174,"thread_start_us":105,"threads_started":1}
I20260621 05:10:40.794129 3598 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260621 05:10:40.800922 9117 catalog_manager.cc:1370] Generated new certificate authority record
I20260621 05:10:40.801429 9117 catalog_manager.cc:1504] Loading token signing keys...
I20260621 05:10:40.807848 9117 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bf799680f38f4eb7a7e763f4b2e6a046: Generated new TSK 0
I20260621 05:10:40.808063 9117 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260621 05:10:40.882886 9107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:10:40.818111 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52044 (local address 127.3.131.190:41839)
0621 05:10:40.818205 (+ 94us) server_negotiation.cc:206] Beginning negotiation
0621 05:10:40.818207 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:10:40.818279 (+ 72us) server_negotiation.cc:407] Connection header received
0621 05:10:40.818349 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:10:40.818349 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:10:40.818370 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:10:40.818435 (+ 65us) server_negotiation.cc:226] Negotiated authn=JWT
0621 05:10:40.818679 (+ 244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:10:40.819279 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:10:40.820447 (+ 1168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:10:40.820693 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:10:40.820758 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:10:40.821033 (+ 275us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0621 05:10:40.882660 (+ 61627us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0621 05:10:40.882828 (+ 168us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:52044: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":69}
W20260621 05:10:40.882992 9107 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:52044: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260621 05:10:40.883078 9132 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 05:10:40.818053 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.131.190:41839 (local address 127.0.0.1:52044)
0621 05:10:40.818201 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0621 05:10:40.818212 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0621 05:10:40.818276 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:10:40.818535 (+ 259us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:10:40.818536 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:10:40.818538 (+ 2us) client_negotiation.cc:189] Negotiated authn=JWT
0621 05:10:40.818603 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:10:40.818605 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:10:40.819372 (+ 767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:10:40.819373 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:10:40.820343 (+ 970us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:10:40.820346 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:10:40.820784 (+ 438us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:10:40.820785 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:10:40.820848 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:10:40.820854 (+ 6us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0621 05:10:40.882815 (+ 61961us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0621 05:10:40.882962 (+ 147us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.3.131.190:41839: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":80,"threads_started":1}
I20260621 05:10:40.883502 3598 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWZOx2_/build/release/bin/kudu with pid 9036
2026-06-21T05:10:40Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (259 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (95 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (98 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782018616648578-3598-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 21 05:10:41 dist-test-slave-vrq1 krb5kdc[9144](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 21 05:10:41 dist-test-slave-vrq1 krb5kdc[9144](info): set up 2 sockets
Jun 21 05:10:41 dist-test-slave-vrq1 krb5kdc[9144](info): commencing operation
krb5kdc: starting...
W20260621 05:10:42.147783 3598 mini_kdc.cc:121] Time spent starting KDC: real 1.052s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782018616648578-3598-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782018616648578-3598-0/krb5kdc/test-user.keytab.
Jun 21 05:10:42 dist-test-slave-vrq1 krb5kdc[9144](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782018642, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-21T05:10:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-21T05:10:42Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.131.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.131.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.131.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782018616648578-3598-0/krb5kdc/kudu_127.3.131.190.keytab.
Entry for principal kudu/127.3.131.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782018616648578-3598-0/krb5kdc/kudu_127.3.131.190.keytab.
I20260621 05:10:42.278563 3598 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWZOx2_/build/release/bin/kudu
/tmp/dist-test-taskWZOx2_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782018616648578-3598-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782018616648578-3598-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782018616648578-3598-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782018616648578-3598-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.131.190:41565
--webserver_interface=127.3.131.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782018616648578-3598-0/krb5kdc/kudu_127.3.131.190.keytab
--principal=kudu/127.3.131.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.131.148:36315
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.131.190:41565
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782018616648578-3598-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782018616648578-3598-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782018616648578-3598-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 05:10:42.362854 9157 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 05:10:42.363147 9157 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 05:10:42.363229 9157 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 05:10:42.364574 9157 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 05:10:42.364667 9157 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 05:10:42.364691 9157 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 05:10:42.364712 9157 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 05:10:42.364741 9157 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 05:10:42.366439 9157 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.131.148:36315
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782018616648578-3598-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782018616648578-3598-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.131.190:41565
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782018616648578-3598-0/krb5kdc/kudu_127.3.131.190.keytab
--principal=kudu/127.3.131.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.131.190:41565
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782018616648578-3598-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.131.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9157
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782018616648578-3598-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 21 Jun 2026 04:16:16 UTC on ccycloud.quasar-xycaha.root.comops.site
build id 4457881
I20260621 05:10:42.366763 9157 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 05:10:42.367017 9157 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260621 05:10:42.372506 9157 server_base.cc:1034] running on GCE node
W20260621 05:10:42.372491 9165 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:10:42.372486 9162 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:10:42.372478 9163 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:10:42.373024 9157 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 05:10:42.373536 9157 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 05:10:42.374809 9157 hybrid_clock.cc:648] HybridClock initialized: now 1782018642374792 us; error 54 us; skew 500 ppm
Jun 21 05:10:42 dist-test-slave-vrq1 krb5kdc[9144](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782018642, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.131.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 05:10:42.377538 9157 init.cc:377] Logged in from keytab as kudu/127.3.131.190@KRBTEST.COM (short username kudu)
I20260621 05:10:42.378093 9157 webserver.cc:460] Webserver started at http://127.3.131.190:38831/ using document root <none> and password file <none>
I20260621 05:10:42.378363 9157 fs_manager.cc:359] Metadata directory not provided
I20260621 05:10:42.378437 9157 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 05:10:42.378592 9157 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 05:10:42.379801 9157 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782018616648578-3598-0/minicluster-data/master-0/data/instance:
uuid: "ee60cf7290ab429bbf089ef7b78cedfc"
format_stamp: "Formatted at 2026-06-21 05:10:42 on dist-test-slave-vrq1"
server_key: "f9a8e3d2e15a2dbac519467e5830c486"
server_key_iv: "7f8e6ec39381c9b060d2c49c52570af7"
server_key_version: "encryptionkey@0"
I20260621 05:10:42.380198 9157 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782018616648578-3598-0/minicluster-data/master-0/wal/instance:
uuid: "ee60cf7290ab429bbf089ef7b78cedfc"
format_stamp: "Formatted at 2026-06-21 05:10:42 on dist-test-slave-vrq1"
server_key: "f9a8e3d2e15a2dbac519467e5830c486"
server_key_iv: "7f8e6ec39381c9b060d2c49c52570af7"
server_key_version: "encryptionkey@0"
I20260621 05:10:42.381645 9157 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260621 05:10:42.382663 9172 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 05:10:42.383106 9157 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260621 05:10:42.383383 9157 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782018616648578-3598-0/minicluster-data/master-0/data,/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782018616648578-3598-0/minicluster-data/master-0/wal
uuid: "ee60cf7290ab429bbf089ef7b78cedfc"
format_stamp: "Formatted at 2026-06-21 05:10:42 on dist-test-slave-vrq1"
server_key: "f9a8e3d2e15a2dbac519467e5830c486"
server_key_iv: "7f8e6ec39381c9b060d2c49c52570af7"
server_key_version: "encryptionkey@0"
I20260621 05:10:42.383473 9157 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782018616648578-3598-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782018616648578-3598-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782018616648578-3598-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 05:10:42.401858 9157 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 05:10:42.404587 9157 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 05:10:42.404770 9157 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 05:10:42.409614 9157 rpc_server.cc:307] RPC server started. Bound to: 127.3.131.190:41565
I20260621 05:10:42.409662 9224 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.131.190:41565 every 8 connection(s)
I20260621 05:10:42.410043 9157 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782018616648578-3598-0/minicluster-data/master-0/data/info.pb
I20260621 05:10:42.410966 9225 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 05:10:42.413445 9225 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ee60cf7290ab429bbf089ef7b78cedfc: Bootstrap starting.
I20260621 05:10:42.414106 9225 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ee60cf7290ab429bbf089ef7b78cedfc: Neither blocks nor log segments found. Creating new log.
I20260621 05:10:42.414409 9225 log.cc:824] T 00000000000000000000000000000000 P ee60cf7290ab429bbf089ef7b78cedfc: Log is configured to *not* fsync() on all Append() calls
I20260621 05:10:42.414484 3598 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWZOx2_/build/release/bin/kudu as pid 9157
I20260621 05:10:42.414630 3598 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782018616648578-3598-0/minicluster-data/master-0/wal/instance
I20260621 05:10:42.414821 3598 external_mini_cluster.cc:1428] Setting key d382c9f8cb700790ef336c54721aeeac
I20260621 05:10:42.415253 9225 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ee60cf7290ab429bbf089ef7b78cedfc: No bootstrap required, opened a new log
I20260621 05:10:42.417084 9225 raft_consensus.cc:348] T 00000000000000000000000000000000 P ee60cf7290ab429bbf089ef7b78cedfc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee60cf7290ab429bbf089ef7b78cedfc" member_type: VOTER last_known_addr { host: "127.3.131.190" port: 41565 } }
I20260621 05:10:42.417263 9225 raft_consensus.cc:374] T 00000000000000000000000000000000 P ee60cf7290ab429bbf089ef7b78cedfc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 05:10:42.417309 9225 raft_consensus.cc:729] T 00000000000000000000000000000000 P ee60cf7290ab429bbf089ef7b78cedfc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ee60cf7290ab429bbf089ef7b78cedfc, State: Initialized, Role: FOLLOWER
I20260621 05:10:42.417439 9225 consensus_queue.cc:260] T 00000000000000000000000000000000 P ee60cf7290ab429bbf089ef7b78cedfc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee60cf7290ab429bbf089ef7b78cedfc" member_type: VOTER last_known_addr { host: "127.3.131.190" port: 41565 } }
I20260621 05:10:42.417541 9225 raft_consensus.cc:388] T 00000000000000000000000000000000 P ee60cf7290ab429bbf089ef7b78cedfc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260621 05:10:42.417585 9225 raft_consensus.cc:482] T 00000000000000000000000000000000 P ee60cf7290ab429bbf089ef7b78cedfc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260621 05:10:42.417644 9225 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ee60cf7290ab429bbf089ef7b78cedfc [term 0 FOLLOWER]: Advancing to term 1
I20260621 05:10:42.418543 9225 raft_consensus.cc:504] T 00000000000000000000000000000000 P ee60cf7290ab429bbf089ef7b78cedfc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee60cf7290ab429bbf089ef7b78cedfc" member_type: VOTER last_known_addr { host: "127.3.131.190" port: 41565 } }
I20260621 05:10:42.418673 9225 leader_election.cc:302] T 00000000000000000000000000000000 P ee60cf7290ab429bbf089ef7b78cedfc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ee60cf7290ab429bbf089ef7b78cedfc; no voters:
I20260621 05:10:42.418877 9225 leader_election.cc:288] T 00000000000000000000000000000000 P ee60cf7290ab429bbf089ef7b78cedfc [CANDIDATE]: Term 1 election: Requested vote from peers
I20260621 05:10:42.418907 9230 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ee60cf7290ab429bbf089ef7b78cedfc [term 1 FOLLOWER]: Leader election won for term 1
I20260621 05:10:42.419023 9230 raft_consensus.cc:686] T 00000000000000000000000000000000 P ee60cf7290ab429bbf089ef7b78cedfc [term 1 LEADER]: Becoming Leader. State: Replica: ee60cf7290ab429bbf089ef7b78cedfc, State: Running, Role: LEADER
I20260621 05:10:42.419111 9230 consensus_queue.cc:237] T 00000000000000000000000000000000 P ee60cf7290ab429bbf089ef7b78cedfc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee60cf7290ab429bbf089ef7b78cedfc" member_type: VOTER last_known_addr { host: "127.3.131.190" port: 41565 } }
I20260621 05:10:42.419166 9225 sys_catalog.cc:564] T 00000000000000000000000000000000 P ee60cf7290ab429bbf089ef7b78cedfc [sys.catalog]: configured and running, proceeding with master startup.
I20260621 05:10:42.419468 9230 sys_catalog.cc:455] T 00000000000000000000000000000000 P ee60cf7290ab429bbf089ef7b78cedfc [sys.catalog]: SysCatalogTable state changed. Reason: New leader ee60cf7290ab429bbf089ef7b78cedfc. Latest consensus state: current_term: 1 leader_uuid: "ee60cf7290ab429bbf089ef7b78cedfc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee60cf7290ab429bbf089ef7b78cedfc" member_type: VOTER last_known_addr { host: "127.3.131.190" port: 41565 } } }
I20260621 05:10:42.419540 9230 sys_catalog.cc:458] T 00000000000000000000000000000000 P ee60cf7290ab429bbf089ef7b78cedfc [sys.catalog]: This master's current role is: LEADER
I20260621 05:10:42.419757 9235 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260621 05:10:42.420317 9235 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260621 05:10:42.420641 9231 sys_catalog.cc:455] T 00000000000000000000000000000000 P ee60cf7290ab429bbf089ef7b78cedfc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ee60cf7290ab429bbf089ef7b78cedfc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee60cf7290ab429bbf089ef7b78cedfc" member_type: VOTER last_known_addr { host: "127.3.131.190" port: 41565 } } }
I20260621 05:10:42.422961 9231 sys_catalog.cc:458] T 00000000000000000000000000000000 P ee60cf7290ab429bbf089ef7b78cedfc [sys.catalog]: This master's current role is: LEADER
I20260621 05:10:42.423115 9235 catalog_manager.cc:1347] Generated new cluster ID: 5529b80292ff48089d505116601c8abc
I20260621 05:10:42.423157 9235 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Jun 21 05:10:42 dist-test-slave-vrq1 krb5kdc[9144](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782018642, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.131.190@KRBTEST.COM
I20260621 05:10:42.428264 9229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:10:42.415478 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45088 (local address 127.3.131.190:41565)
0621 05:10:42.415683 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0621 05:10:42.415695 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0621 05:10:42.415742 (+ 47us) server_negotiation.cc:407] Connection header received
0621 05:10:42.417082 (+ 1340us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:10:42.417091 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:10:42.417169 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:10:42.417381 (+ 212us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:10:42.418132 (+ 751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:10:42.419372 (+ 1240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:10:42.420421 (+ 1049us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:10:42.420979 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:10:42.421104 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:10:42.424054 (+ 2950us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:10:42.424059 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:10:42.424059 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:10:42.424073 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:10:42.426197 (+ 2124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:10:42.426668 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:10:42.426671 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:10:42.426672 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:10:42.426896 (+ 224us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:10:42.427233 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:10:42.427238 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:10:42.427238 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:10:42.427583 (+ 345us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:10:42.427769 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0621 05:10:42.427889 (+ 120us) server_negotiation.cc:299] Negotiation successful
0621 05:10:42.428038 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":147,"thread_start_us":91,"threads_started":1}
I20260621 05:10:42.430979 9235 catalog_manager.cc:1370] Generated new certificate authority record
I20260621 05:10:42.431561 9235 catalog_manager.cc:1504] Loading token signing keys...
I20260621 05:10:42.437327 9235 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ee60cf7290ab429bbf089ef7b78cedfc: Generated new TSK 0
I20260621 05:10:42.437557 9235 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.3.131.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.131.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.131.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782018616648578-3598-0/krb5kdc/kudu_127.3.131.129.keytab.
Entry for principal kudu/127.3.131.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782018616648578-3598-0/krb5kdc/kudu_127.3.131.129.keytab.
I20260621 05:10:42.471455 3598 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWZOx2_/build/release/bin/kudu
/tmp/dist-test-taskWZOx2_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782018616648578-3598-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782018616648578-3598-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782018616648578-3598-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782018616648578-3598-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.131.129:0
--local_ip_for_outbound_sockets=127.3.131.129
--webserver_interface=127.3.131.129
--webserver_port=0
--tserver_master_addrs=127.3.131.190:41565
--keytab_file=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782018616648578-3598-0/krb5kdc/kudu_127.3.131.129.keytab
--principal=kudu/127.3.131.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.131.148:36315
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782018616648578-3598-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782018616648578-3598-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782018616648578-3598-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 05:10:42.568550 9251 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 05:10:42.568786 9251 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 05:10:42.568816 9251 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 05:10:42.570154 9251 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 05:10:42.570220 9251 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 05:10:42.570253 9251 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.131.129
I20260621 05:10:42.572005 9251 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.131.148:36315
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782018616648578-3598-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782018616648578-3598-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782018616648578-3598-0/krb5kdc/kudu_127.3.131.129.keytab
--principal=kudu/127.3.131.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.131.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782018616648578-3598-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.131.129
--webserver_port=0
--tserver_master_addrs=127.3.131.190:41565
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9251
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.131.129
--log_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782018616648578-3598-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 21 Jun 2026 04:16:16 UTC on ccycloud.quasar-xycaha.root.comops.site
build id 4457881
I20260621 05:10:42.572235 9251 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 05:10:42.572436 9251 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 05:10:42.578104 9257 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:10:42.578109 9256 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:10:42.578109 9259 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:10:42.578625 9251 server_base.cc:1034] running on GCE node
I20260621 05:10:42.579027 9251 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 05:10:42.579411 9251 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 05:10:42.580600 9251 hybrid_clock.cc:648] HybridClock initialized: now 1782018642580558 us; error 59 us; skew 500 ppm
Jun 21 05:10:42 dist-test-slave-vrq1 krb5kdc[9144](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782018642, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.131.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 05:10:42.583768 9251 init.cc:377] Logged in from keytab as kudu/127.3.131.129@KRBTEST.COM (short username kudu)
I20260621 05:10:42.584278 9251 webserver.cc:460] Webserver started at http://127.3.131.129:39393/ using document root <none> and password file <none>
I20260621 05:10:42.584546 9251 fs_manager.cc:359] Metadata directory not provided
I20260621 05:10:42.584596 9251 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 05:10:42.584738 9251 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 05:10:42.585906 9251 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782018616648578-3598-0/minicluster-data/ts-0/data/instance:
uuid: "44eab50f52fe43428198103b010d228e"
format_stamp: "Formatted at 2026-06-21 05:10:42 on dist-test-slave-vrq1"
server_key: "81dec1973c396179fb68ca049a527dfe"
server_key_iv: "1eccd1fbb34cf59db5cf55cfa20b353e"
server_key_version: "encryptionkey@0"
I20260621 05:10:42.586360 9251 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782018616648578-3598-0/minicluster-data/ts-0/wal/instance:
uuid: "44eab50f52fe43428198103b010d228e"
format_stamp: "Formatted at 2026-06-21 05:10:42 on dist-test-slave-vrq1"
server_key: "81dec1973c396179fb68ca049a527dfe"
server_key_iv: "1eccd1fbb34cf59db5cf55cfa20b353e"
server_key_version: "encryptionkey@0"
I20260621 05:10:42.587991 9251 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260621 05:10:42.588917 9266 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 05:10:42.589201 9251 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260621 05:10:42.589325 9251 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782018616648578-3598-0/minicluster-data/ts-0/data,/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782018616648578-3598-0/minicluster-data/ts-0/wal
uuid: "44eab50f52fe43428198103b010d228e"
format_stamp: "Formatted at 2026-06-21 05:10:42 on dist-test-slave-vrq1"
server_key: "81dec1973c396179fb68ca049a527dfe"
server_key_iv: "1eccd1fbb34cf59db5cf55cfa20b353e"
server_key_version: "encryptionkey@0"
I20260621 05:10:42.589413 9251 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782018616648578-3598-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782018616648578-3598-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782018616648578-3598-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 05:10:42.606966 9251 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 05:10:42.609889 9251 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 05:10:42.610091 9251 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 05:10:42.610592 9251 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 05:10:42.611006 9251 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 05:10:42.611044 9251 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 05:10:42.611069 9251 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 05:10:42.611083 9251 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 05:10:42.618017 9251 rpc_server.cc:307] RPC server started. Bound to: 127.3.131.129:33661
I20260621 05:10:42.618104 9379 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.131.129:33661 every 8 connection(s)
I20260621 05:10:42.618463 9251 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782018616648578-3598-0/minicluster-data/ts-0/data/info.pb
Jun 21 05:10:42 dist-test-slave-vrq1 krb5kdc[9144](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782018642, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.131.129@KRBTEST.COM for kudu/127.3.131.190@KRBTEST.COM
I20260621 05:10:42.627923 3598 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWZOx2_/build/release/bin/kudu as pid 9251
I20260621 05:10:42.628048 3598 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782018616648578-3598-0/minicluster-data/ts-0/wal/instance
I20260621 05:10:42.628260 3598 external_mini_cluster.cc:1428] Setting key abf4ebbd16134b53d142e02eb07857d4
I20260621 05:10:42.630466 9229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:10:42.619408 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.131.129:48929 (local address 127.3.131.190:41565)
0621 05:10:42.619462 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0621 05:10:42.619464 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:10:42.619783 (+ 319us) server_negotiation.cc:407] Connection header received
0621 05:10:42.620950 (+ 1167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:10:42.620952 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:10:42.620965 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:10:42.621042 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:10:42.621834 (+ 792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:10:42.622374 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:10:42.623688 (+ 1314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:10:42.623905 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:10:42.623973 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:10:42.626555 (+ 2582us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:10:42.626557 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:10:42.626557 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:10:42.626571 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:10:42.628513 (+ 1942us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:10:42.629214 (+ 701us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:10:42.629216 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:10:42.629216 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:10:42.629294 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:10:42.629596 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:10:42.629598 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:10:42.629598 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:10:42.629767 (+ 169us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:10:42.629853 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0621 05:10:42.630209 (+ 356us) server_negotiation.cc:299] Negotiation successful
0621 05:10:42.630317 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260621 05:10:42.630623 9382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:10:42.619421 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.131.190:41565 (local address 127.3.131.129:48929)
0621 05:10:42.619639 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0621 05:10:42.619679 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0621 05:10:42.620797 (+ 1118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:10:42.621156 (+ 359us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:10:42.621160 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:10:42.621436 (+ 276us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:10:42.621677 (+ 241us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:10:42.621682 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:10:42.622574 (+ 892us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:10:42.622575 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:10:42.623555 (+ 980us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:10:42.623558 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:10:42.624030 (+ 472us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:10:42.624031 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:10:42.624178 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:10:42.624849 (+ 671us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 05:10:42.624859 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:10:42.626400 (+ 1541us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 05:10:42.628606 (+ 2206us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:10:42.628617 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:10:42.628619 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:10:42.629064 (+ 445us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:10:42.629373 (+ 309us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:10:42.629374 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:10:42.629374 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:10:42.629498 (+ 124us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:10:42.629896 (+ 398us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 05:10:42.629903 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 05:10:42.630055 (+ 152us) client_negotiation.cc:769] Sending connection context
0621 05:10:42.630213 (+ 158us) client_negotiation.cc:240] Negotiation successful
0621 05:10:42.630374 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":167,"thread_start_us":96,"threads_started":1}
I20260621 05:10:42.631479 9380 heartbeater.cc:347] Connected to a master server at 127.3.131.190:41565
I20260621 05:10:42.631613 9380 heartbeater.cc:464] Registering TS with master...
I20260621 05:10:42.631843 9380 heartbeater.cc:511] Master 127.3.131.190:41565 requested a full tablet report, sending...
I20260621 05:10:42.632388 9189 ts_manager.cc:194] Registered new tserver with Master: 44eab50f52fe43428198103b010d228e (127.3.131.129:33661)
I20260621 05:10:42.633055 9189 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.131.129@KRBTEST.COM'} at 127.3.131.129:48929
WARNING: no policy specified for kudu/127.3.131.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.131.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.131.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782018616648578-3598-0/krb5kdc/kudu_127.3.131.130.keytab.
Entry for principal kudu/127.3.131.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782018616648578-3598-0/krb5kdc/kudu_127.3.131.130.keytab.
I20260621 05:10:42.658910 3598 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWZOx2_/build/release/bin/kudu
/tmp/dist-test-taskWZOx2_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782018616648578-3598-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782018616648578-3598-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782018616648578-3598-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782018616648578-3598-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.131.130:0
--local_ip_for_outbound_sockets=127.3.131.130
--webserver_interface=127.3.131.130
--webserver_port=0
--tserver_master_addrs=127.3.131.190:41565
--keytab_file=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782018616648578-3598-0/krb5kdc/kudu_127.3.131.130.keytab
--principal=kudu/127.3.131.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.131.148:36315
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782018616648578-3598-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782018616648578-3598-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782018616648578-3598-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 05:10:42.747100 9385 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 05:10:42.747390 9385 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 05:10:42.747421 9385 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 05:10:42.748687 9385 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 05:10:42.748745 9385 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 05:10:42.748776 9385 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.131.130
I20260621 05:10:42.750584 9385 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.131.148:36315
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782018616648578-3598-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782018616648578-3598-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782018616648578-3598-0/krb5kdc/kudu_127.3.131.130.keytab
--principal=kudu/127.3.131.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.131.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782018616648578-3598-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.131.130
--webserver_port=0
--tserver_master_addrs=127.3.131.190:41565
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9385
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.131.130
--log_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782018616648578-3598-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 21 Jun 2026 04:16:16 UTC on ccycloud.quasar-xycaha.root.comops.site
build id 4457881
I20260621 05:10:42.750836 9385 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 05:10:42.751029 9385 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 05:10:42.756534 9391 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:10:42.756574 9385 server_base.cc:1034] running on GCE node
W20260621 05:10:42.756567 9390 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:10:42.756534 9393 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:10:42.757081 9385 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 05:10:42.757404 9385 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 05:10:42.758615 9385 hybrid_clock.cc:648] HybridClock initialized: now 1782018642758584 us; error 60 us; skew 500 ppm
Jun 21 05:10:42 dist-test-slave-vrq1 krb5kdc[9144](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782018642, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.131.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 05:10:42.761873 9385 init.cc:377] Logged in from keytab as kudu/127.3.131.130@KRBTEST.COM (short username kudu)
I20260621 05:10:42.762621 9385 webserver.cc:460] Webserver started at http://127.3.131.130:34157/ using document root <none> and password file <none>
I20260621 05:10:42.762934 9385 fs_manager.cc:359] Metadata directory not provided
I20260621 05:10:42.763013 9385 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 05:10:42.763262 9385 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 05:10:42.764437 9385 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782018616648578-3598-0/minicluster-data/ts-1/data/instance:
uuid: "df6822b7d158403cb55b0d62ae7415e2"
format_stamp: "Formatted at 2026-06-21 05:10:42 on dist-test-slave-vrq1"
server_key: "cf716244903ac89f744a955a234cd657"
server_key_iv: "c4cea5739a03b44956e6c86106358307"
server_key_version: "encryptionkey@0"
I20260621 05:10:42.764832 9385 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782018616648578-3598-0/minicluster-data/ts-1/wal/instance:
uuid: "df6822b7d158403cb55b0d62ae7415e2"
format_stamp: "Formatted at 2026-06-21 05:10:42 on dist-test-slave-vrq1"
server_key: "cf716244903ac89f744a955a234cd657"
server_key_iv: "c4cea5739a03b44956e6c86106358307"
server_key_version: "encryptionkey@0"
I20260621 05:10:42.766683 9385 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.003s sys 0.000s
I20260621 05:10:42.767834 9400 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 05:10:42.768087 9385 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260621 05:10:42.768381 9385 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782018616648578-3598-0/minicluster-data/ts-1/data,/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782018616648578-3598-0/minicluster-data/ts-1/wal
uuid: "df6822b7d158403cb55b0d62ae7415e2"
format_stamp: "Formatted at 2026-06-21 05:10:42 on dist-test-slave-vrq1"
server_key: "cf716244903ac89f744a955a234cd657"
server_key_iv: "c4cea5739a03b44956e6c86106358307"
server_key_version: "encryptionkey@0"
I20260621 05:10:42.768489 9385 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782018616648578-3598-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782018616648578-3598-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782018616648578-3598-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 05:10:42.804392 9385 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 05:10:42.807657 9385 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 05:10:42.807852 9385 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 05:10:42.808123 9385 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 05:10:42.808466 9385 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 05:10:42.808537 9385 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 05:10:42.808583 9385 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 05:10:42.808612 9385 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 05:10:42.815295 9385 rpc_server.cc:307] RPC server started. Bound to: 127.3.131.130:36451
I20260621 05:10:42.815356 9513 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.131.130:36451 every 8 connection(s)
I20260621 05:10:42.815747 9385 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782018616648578-3598-0/minicluster-data/ts-1/data/info.pb
Jun 21 05:10:42 dist-test-slave-vrq1 krb5kdc[9144](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782018642, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.131.130@KRBTEST.COM for kudu/127.3.131.190@KRBTEST.COM
I20260621 05:10:42.825747 3598 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWZOx2_/build/release/bin/kudu as pid 9385
I20260621 05:10:42.825870 3598 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782018616648578-3598-0/minicluster-data/ts-1/wal/instance
I20260621 05:10:42.826083 3598 external_mini_cluster.cc:1428] Setting key e55b486eba10e2b55e60bf700966fc7d
I20260621 05:10:42.828753 9229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:10:42.816676 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.131.130:58189 (local address 127.3.131.190:41565)
0621 05:10:42.816788 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0621 05:10:42.816790 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:10:42.817044 (+ 254us) server_negotiation.cc:407] Connection header received
0621 05:10:42.818100 (+ 1056us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:10:42.818102 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:10:42.818153 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:10:42.818221 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:10:42.819018 (+ 797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:10:42.819830 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:10:42.821112 (+ 1282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:10:42.821353 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:10:42.821427 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:10:42.824337 (+ 2910us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:10:42.824339 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:10:42.824339 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:10:42.824348 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:10:42.826656 (+ 2308us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:10:42.827318 (+ 662us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:10:42.827320 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:10:42.827320 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:10:42.827397 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:10:42.827755 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:10:42.827756 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:10:42.827757 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:10:42.828043 (+ 286us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:10:42.828130 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0621 05:10:42.828483 (+ 353us) server_negotiation.cc:299] Negotiation successful
0621 05:10:42.828604 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260621 05:10:42.828825 9516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:10:42.816635 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.131.190:41565 (local address 127.3.131.130:58189)
0621 05:10:42.816872 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0621 05:10:42.816922 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0621 05:10:42.817977 (+ 1055us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:10:42.818286 (+ 309us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:10:42.818290 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:10:42.818687 (+ 397us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:10:42.818910 (+ 223us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:10:42.818916 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:10:42.819951 (+ 1035us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:10:42.819952 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:10:42.821005 (+ 1053us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:10:42.821009 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:10:42.821465 (+ 456us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:10:42.821466 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:10:42.821590 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:10:42.822298 (+ 708us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 05:10:42.822308 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:10:42.824082 (+ 1774us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 05:10:42.826767 (+ 2685us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:10:42.826778 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:10:42.826781 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:10:42.827224 (+ 443us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:10:42.827517 (+ 293us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:10:42.827518 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:10:42.827519 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:10:42.827665 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:10:42.828171 (+ 506us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 05:10:42.828178 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 05:10:42.828331 (+ 153us) client_negotiation.cc:769] Sending connection context
0621 05:10:42.828437 (+ 106us) client_negotiation.cc:240] Negotiation successful
0621 05:10:42.828599 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":156,"threads_started":1}
I20260621 05:10:42.829756 9514 heartbeater.cc:347] Connected to a master server at 127.3.131.190:41565
I20260621 05:10:42.829897 9514 heartbeater.cc:464] Registering TS with master...
I20260621 05:10:42.830158 9514 heartbeater.cc:511] Master 127.3.131.190:41565 requested a full tablet report, sending...
I20260621 05:10:42.830624 9189 ts_manager.cc:194] Registered new tserver with Master: df6822b7d158403cb55b0d62ae7415e2 (127.3.131.130:36451)
I20260621 05:10:42.831060 9189 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.131.130@KRBTEST.COM'} at 127.3.131.130:58189
WARNING: no policy specified for kudu/127.3.131.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.131.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.131.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782018616648578-3598-0/krb5kdc/kudu_127.3.131.131.keytab.
Entry for principal kudu/127.3.131.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782018616648578-3598-0/krb5kdc/kudu_127.3.131.131.keytab.
I20260621 05:10:42.859627 3598 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWZOx2_/build/release/bin/kudu
/tmp/dist-test-taskWZOx2_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782018616648578-3598-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782018616648578-3598-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782018616648578-3598-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782018616648578-3598-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.131.131:0
--local_ip_for_outbound_sockets=127.3.131.131
--webserver_interface=127.3.131.131
--webserver_port=0
--tserver_master_addrs=127.3.131.190:41565
--keytab_file=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782018616648578-3598-0/krb5kdc/kudu_127.3.131.131.keytab
--principal=kudu/127.3.131.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.131.148:36315
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782018616648578-3598-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782018616648578-3598-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782018616648578-3598-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 05:10:42.947338 9519 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 05:10:42.947589 9519 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 05:10:42.947633 9519 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 05:10:42.948887 9519 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 05:10:42.948959 9519 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 05:10:42.948999 9519 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.131.131
I20260621 05:10:42.950665 9519 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.131.148:36315
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782018616648578-3598-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782018616648578-3598-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782018616648578-3598-0/krb5kdc/kudu_127.3.131.131.keytab
--principal=kudu/127.3.131.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.131.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782018616648578-3598-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.131.131
--webserver_port=0
--tserver_master_addrs=127.3.131.190:41565
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9519
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.131.131
--log_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782018616648578-3598-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 21 Jun 2026 04:16:16 UTC on ccycloud.quasar-xycaha.root.comops.site
build id 4457881
I20260621 05:10:42.950955 9519 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 05:10:42.951256 9519 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 05:10:42.956684 9527 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:10:42.956681 9524 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:10:42.956674 9525 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:10:42.957463 9519 server_base.cc:1034] running on GCE node
I20260621 05:10:42.957756 9519 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 05:10:42.958047 9519 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 05:10:42.959307 9519 hybrid_clock.cc:648] HybridClock initialized: now 1782018642959297 us; error 55 us; skew 500 ppm
Jun 21 05:10:42 dist-test-slave-vrq1 krb5kdc[9144](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782018642, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.131.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 05:10:42.962402 9519 init.cc:377] Logged in from keytab as kudu/127.3.131.131@KRBTEST.COM (short username kudu)
I20260621 05:10:42.963040 9519 webserver.cc:460] Webserver started at http://127.3.131.131:34795/ using document root <none> and password file <none>
I20260621 05:10:42.963339 9519 fs_manager.cc:359] Metadata directory not provided
I20260621 05:10:42.963413 9519 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 05:10:42.963555 9519 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 05:10:42.964646 9519 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782018616648578-3598-0/minicluster-data/ts-2/data/instance:
uuid: "8e80f8df321a4118a6894bbcc10b179c"
format_stamp: "Formatted at 2026-06-21 05:10:42 on dist-test-slave-vrq1"
server_key: "494bc9fdb1897f8063992dd44b44fc73"
server_key_iv: "a9c486a36f8136c8250b8dfd1d994660"
server_key_version: "encryptionkey@0"
I20260621 05:10:42.965040 9519 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782018616648578-3598-0/minicluster-data/ts-2/wal/instance:
uuid: "8e80f8df321a4118a6894bbcc10b179c"
format_stamp: "Formatted at 2026-06-21 05:10:42 on dist-test-slave-vrq1"
server_key: "494bc9fdb1897f8063992dd44b44fc73"
server_key_iv: "a9c486a36f8136c8250b8dfd1d994660"
server_key_version: "encryptionkey@0"
I20260621 05:10:42.966542 9519 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260621 05:10:42.967464 9534 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 05:10:42.967679 9519 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260621 05:10:42.967784 9519 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782018616648578-3598-0/minicluster-data/ts-2/data,/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782018616648578-3598-0/minicluster-data/ts-2/wal
uuid: "8e80f8df321a4118a6894bbcc10b179c"
format_stamp: "Formatted at 2026-06-21 05:10:42 on dist-test-slave-vrq1"
server_key: "494bc9fdb1897f8063992dd44b44fc73"
server_key_iv: "a9c486a36f8136c8250b8dfd1d994660"
server_key_version: "encryptionkey@0"
I20260621 05:10:42.967876 9519 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782018616648578-3598-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782018616648578-3598-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782018616648578-3598-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 05:10:42.995011 9519 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 05:10:42.997720 9519 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 05:10:42.997905 9519 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 05:10:42.998183 9519 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 05:10:42.998524 9519 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 05:10:42.998580 9519 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 05:10:42.998623 9519 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 05:10:42.998648 9519 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 05:10:43.004958 9519 rpc_server.cc:307] RPC server started. Bound to: 127.3.131.131:35325
I20260621 05:10:43.004997 9647 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.131.131:35325 every 8 connection(s)
I20260621 05:10:43.005347 9519 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782018616648578-3598-0/minicluster-data/ts-2/data/info.pb
I20260621 05:10:43.005577 3598 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWZOx2_/build/release/bin/kudu as pid 9519
I20260621 05:10:43.005671 3598 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782018616648578-3598-0/minicluster-data/ts-2/wal/instance
I20260621 05:10:43.005900 3598 external_mini_cluster.cc:1428] Setting key 6361e3d79ba355aa49b307fe616ed659
Jun 21 05:10:43 dist-test-slave-vrq1 krb5kdc[9144](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782018642, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.131.131@KRBTEST.COM for kudu/127.3.131.190@KRBTEST.COM
I20260621 05:10:43.015996 9229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:10:43.006185 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.131.131:40019 (local address 127.3.131.190:41565)
0621 05:10:43.006241 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0621 05:10:43.006244 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0621 05:10:43.006523 (+ 279us) server_negotiation.cc:407] Connection header received
0621 05:10:43.007721 (+ 1198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:10:43.007722 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:10:43.007733 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:10:43.007780 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:10:43.008415 (+ 635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:10:43.008917 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:10:43.009987 (+ 1070us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:10:43.010207 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:10:43.010278 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:10:43.012784 (+ 2506us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:10:43.012786 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:10:43.012787 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:10:43.012795 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:10:43.014319 (+ 1524us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:10:43.014869 (+ 550us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:10:43.014871 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:10:43.014872 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:10:43.014940 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:10:43.015285 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:10:43.015287 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:10:43.015287 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:10:43.015419 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:10:43.015501 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0621 05:10:43.015806 (+ 305us) server_negotiation.cc:299] Negotiation successful
0621 05:10:43.015903 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260621 05:10:43.016207 9650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:10:43.006183 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.131.190:41565 (local address 127.3.131.131:40019)
0621 05:10:43.006367 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0621 05:10:43.006443 (+ 76us) client_negotiation.cc:174] Beginning negotiation
0621 05:10:43.007614 (+ 1171us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:10:43.007848 (+ 234us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:10:43.007851 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:10:43.008112 (+ 261us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:10:43.008317 (+ 205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:10:43.008321 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:10:43.009000 (+ 679us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:10:43.009001 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:10:43.009884 (+ 883us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:10:43.009886 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:10:43.010318 (+ 432us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:10:43.010319 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:10:43.010442 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:10:43.010993 (+ 551us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 05:10:43.011001 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:10:43.012608 (+ 1607us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 05:10:43.014501 (+ 1893us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:10:43.014509 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:10:43.014512 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:10:43.014783 (+ 271us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:10:43.015010 (+ 227us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:10:43.015011 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:10:43.015012 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:10:43.015132 (+ 120us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:10:43.015491 (+ 359us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 05:10:43.015497 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 05:10:43.015656 (+ 159us) client_negotiation.cc:769] Sending connection context
0621 05:10:43.015771 (+ 115us) client_negotiation.cc:240] Negotiation successful
0621 05:10:43.015943 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":99,"threads_started":1}
I20260621 05:10:43.016765 9648 heartbeater.cc:347] Connected to a master server at 127.3.131.190:41565
I20260621 05:10:43.016884 9648 heartbeater.cc:464] Registering TS with master...
I20260621 05:10:43.017093 9648 heartbeater.cc:511] Master 127.3.131.190:41565 requested a full tablet report, sending...
I20260621 05:10:43.017486 9189 ts_manager.cc:194] Registered new tserver with Master: 8e80f8df321a4118a6894bbcc10b179c (127.3.131.131:35325)
I20260621 05:10:43.017877 9189 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.131.131@KRBTEST.COM'} at 127.3.131.131:40019
I20260621 05:10:43.017864 3598 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782018616648578-3598-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782018616648578-3598-0/krb5kdc/test-admin.keytab.
Jun 21 05:10:43 dist-test-slave-vrq1 krb5kdc[9144](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782018643, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 05:10:43.037571 3598 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260621 05:10:43.039559 9655 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 05:10:43.038594 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50550 (local address 127.3.131.129:33661)
0621 05:10:43.038782 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0621 05:10:43.038787 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 05:10:43.038799 (+ 12us) server_negotiation.cc:407] Connection header received
0621 05:10:43.038825 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:10:43.038827 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:10:43.038841 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:10:43.038912 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:10:43.039450 (+ 538us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50550: BlockingRecv error: recv got EOF from 127.0.0.1:50550 (error 108)
Metrics: {"server-negotiator.queue_time_us":150,"thread_start_us":100,"threads_started":1}
W20260621 05:10:43.039597 3598 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782018616648578-3598-0/krb5kdc/krb5cc' of type FILE
Jun 21 05:10:43 dist-test-slave-vrq1 krb5kdc[9144](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782018643, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 05:10:43.040709 3598 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Jun 21 05:10:43 dist-test-slave-vrq1 krb5kdc[9144](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782018643, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 05:10:43.041775 3598 init.cc:295] Successfully reacquired a new kerberos TGT
Jun 21 05:10:43 dist-test-slave-vrq1 krb5kdc[9144](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782018643, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.131.129@KRBTEST.COM
I20260621 05:10:43.050500 9655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:10:43.042426 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50562 (local address 127.3.131.129:33661)
0621 05:10:43.042499 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0621 05:10:43.042501 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:10:43.042511 (+ 10us) server_negotiation.cc:407] Connection header received
0621 05:10:43.042606 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:10:43.042607 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:10:43.042618 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:10:43.042724 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:10:43.043110 (+ 386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:10:43.043929 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:10:43.044682 (+ 753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:10:43.045020 (+ 338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:10:43.045101 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:10:43.047332 (+ 2231us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:10:43.047342 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:10:43.047342 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:10:43.047350 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:10:43.049064 (+ 1714us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:10:43.049545 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:10:43.049549 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:10:43.049549 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:10:43.049612 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:10:43.049851 (+ 239us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:10:43.049852 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:10:43.049853 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:10:43.050020 (+ 167us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:10:43.050192 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0621 05:10:43.050283 (+ 91us) server_negotiation.cc:299] Negotiation successful
0621 05:10:43.050373 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
W20260621 05:10:43.052800 9655 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 05:10:43.052065 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50574 (local address 127.3.131.129:33661)
0621 05:10:43.052167 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0621 05:10:43.052169 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:10:43.052264 (+ 95us) server_negotiation.cc:407] Connection header received
0621 05:10:43.052327 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:10:43.052328 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:10:43.052339 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:10:43.052409 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:10:43.052752 (+ 343us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50574: BlockingRecv error: recv got EOF from 127.0.0.1:50574 (error 108)
Metrics: {"server-negotiator.queue_time_us":73}
W20260621 05:10:43.053097 3598 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782018616648578-3598-0/krb5kdc/krb5cc' of type FILE
Jun 21 05:10:43 dist-test-slave-vrq1 krb5kdc[9144](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782018643, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 05:10:43.054142 3598 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Jun 21 05:10:43 dist-test-slave-vrq1 krb5kdc[9144](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782018643, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 05:10:43.054991 3598 init.cc:295] Successfully reacquired a new kerberos TGT
Jun 21 05:10:43 dist-test-slave-vrq1 krb5kdc[9144](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782018643, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.131.129@KRBTEST.COM
I20260621 05:10:43.062927 9655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:10:43.055662 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50576 (local address 127.3.131.129:33661)
0621 05:10:43.055734 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0621 05:10:43.055736 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:10:43.055746 (+ 10us) server_negotiation.cc:407] Connection header received
0621 05:10:43.055796 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:10:43.055797 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:10:43.055812 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:10:43.055872 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:10:43.056267 (+ 395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:10:43.056822 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:10:43.057545 (+ 723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:10:43.057751 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:10:43.057824 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:10:43.060301 (+ 2477us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:10:43.060302 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:10:43.060303 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:10:43.060311 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:10:43.061674 (+ 1363us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:10:43.062072 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:10:43.062073 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:10:43.062073 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:10:43.062171 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:10:43.062376 (+ 205us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:10:43.062377 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:10:43.062378 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:10:43.062483 (+ 105us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:10:43.062590 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0621 05:10:43.062720 (+ 130us) server_negotiation.cc:299] Negotiation successful
0621 05:10:43.062808 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
W20260621 05:10:43.064728 9655 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 05:10:43.063996 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50588 (local address 127.3.131.129:33661)
0621 05:10:43.064035 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0621 05:10:43.064037 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:10:43.064318 (+ 281us) server_negotiation.cc:407] Connection header received
0621 05:10:43.064341 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:10:43.064342 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:10:43.064352 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:10:43.064396 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:10:43.064681 (+ 285us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50588: BlockingRecv error: recv got EOF from 127.0.0.1:50588 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
W20260621 05:10:43.064807 3598 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782018616648578-3598-0/krb5kdc/krb5cc' of type FILE
Jun 21 05:10:43 dist-test-slave-vrq1 krb5kdc[9144](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782018643, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 05:10:43.065830 3598 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Jun 21 05:10:43 dist-test-slave-vrq1 krb5kdc[9144](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782018643, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 05:10:43.066743 3598 init.cc:295] Successfully reacquired a new kerberos TGT
Jun 21 05:10:43 dist-test-slave-vrq1 krb5kdc[9144](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782018643, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.131.129@KRBTEST.COM
I20260621 05:10:43.074856 9655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:10:43.067290 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50600 (local address 127.3.131.129:33661)
0621 05:10:43.067327 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0621 05:10:43.067329 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:10:43.067406 (+ 77us) server_negotiation.cc:407] Connection header received
0621 05:10:43.067481 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:10:43.067482 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:10:43.067492 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:10:43.067564 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:10:43.067964 (+ 400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:10:43.068479 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:10:43.069188 (+ 709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:10:43.069381 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:10:43.069457 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:10:43.071830 (+ 2373us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:10:43.071831 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:10:43.071831 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:10:43.071839 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:10:43.073497 (+ 1658us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:10:43.073944 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:10:43.073945 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:10:43.073946 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:10:43.074021 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:10:43.074264 (+ 243us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:10:43.074265 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:10:43.074266 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:10:43.074395 (+ 129us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:10:43.074538 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0621 05:10:43.074651 (+ 113us) server_negotiation.cc:299] Negotiation successful
0621 05:10:43.074737 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260621 05:10:43.075683 3598 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWZOx2_/build/release/bin/kudu with pid 9251
I20260621 05:10:43.080729 3598 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWZOx2_/build/release/bin/kudu with pid 9385
I20260621 05:10:43.085528 3598 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWZOx2_/build/release/bin/kudu with pid 9519
I20260621 05:10:43.090097 3598 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWZOx2_/build/release/bin/kudu with pid 9157
2026-06-21T05:10:43Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (2015 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782018616648578-3598-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 21 05:10:43 dist-test-slave-vrq1 krb5kdc[9668](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 21 05:10:43 dist-test-slave-vrq1 krb5kdc[9668](info): set up 2 sockets
Jun 21 05:10:43 dist-test-slave-vrq1 krb5kdc[9668](info): commencing operation
krb5kdc: starting...
W20260621 05:10:44.171804 3598 mini_kdc.cc:121] Time spent starting KDC: real 1.060s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782018616648578-3598-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782018616648578-3598-0/krb5kdc/test-user.keytab.
Jun 21 05:10:44 dist-test-slave-vrq1 krb5kdc[9668](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782018644, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-21T05:10:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-21T05:10:44Z Disabled control of system clock
WARNING: no policy specified for oryx/127.3.131.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.3.131.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.3.131.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782018616648578-3598-0/krb5kdc/oryx_127.3.131.190.keytab.
Entry for principal oryx/127.3.131.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782018616648578-3598-0/krb5kdc/oryx_127.3.131.190.keytab.
I20260621 05:10:44.299468 3598 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWZOx2_/build/release/bin/kudu
/tmp/dist-test-taskWZOx2_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782018616648578-3598-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782018616648578-3598-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782018616648578-3598-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782018616648578-3598-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.131.190:33159
--webserver_interface=127.3.131.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782018616648578-3598-0/krb5kdc/oryx_127.3.131.190.keytab
--principal=oryx/127.3.131.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.131.148:35089
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.131.190:33159
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782018616648578-3598-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782018616648578-3598-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782018616648578-3598-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 05:10:44.382071 9681 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 05:10:44.382360 9681 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 05:10:44.382402 9681 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 05:10:44.383747 9681 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 05:10:44.383821 9681 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 05:10:44.383845 9681 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 05:10:44.383867 9681 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 05:10:44.383894 9681 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 05:10:44.385453 9681 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.131.148:35089
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782018616648578-3598-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782018616648578-3598-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.131.190:33159
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782018616648578-3598-0/krb5kdc/oryx_127.3.131.190.keytab
--principal=oryx/127.3.131.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.131.190:33159
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782018616648578-3598-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.131.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9681
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782018616648578-3598-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 21 Jun 2026 04:16:16 UTC on ccycloud.quasar-xycaha.root.comops.site
build id 4457881
I20260621 05:10:44.385773 9681 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 05:10:44.386020 9681 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 05:10:44.391250 9687 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:10:44.391254 9689 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:10:44.391251 9686 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:10:44.391672 9681 server_base.cc:1034] running on GCE node
I20260621 05:10:44.391965 9681 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 05:10:44.392274 9681 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 05:10:44.393457 9681 hybrid_clock.cc:648] HybridClock initialized: now 1782018644393443 us; error 29 us; skew 500 ppm
Jun 21 05:10:44 dist-test-slave-vrq1 krb5kdc[9668](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782018644, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.131.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 05:10:44.399315 9681 init.cc:377] Logged in from keytab as oryx/127.3.131.190@KRBTEST.COM (short username oryx)
I20260621 05:10:44.399811 9681 webserver.cc:460] Webserver started at http://127.3.131.190:45039/ using document root <none> and password file <none>
I20260621 05:10:44.400053 9681 fs_manager.cc:359] Metadata directory not provided
I20260621 05:10:44.400170 9681 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 05:10:44.400321 9681 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 05:10:44.401443 9681 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782018616648578-3598-0/minicluster-data/master-0/data/instance:
uuid: "08adad2fec7f4618bfbb0504365c5ecc"
format_stamp: "Formatted at 2026-06-21 05:10:44 on dist-test-slave-vrq1"
server_key: "34adf19beb442973c9845596f895dc41"
server_key_iv: "621b4406c77566e9e32d4c79f1680fad"
server_key_version: "encryptionkey@0"
I20260621 05:10:44.401819 9681 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782018616648578-3598-0/minicluster-data/master-0/wal/instance:
uuid: "08adad2fec7f4618bfbb0504365c5ecc"
format_stamp: "Formatted at 2026-06-21 05:10:44 on dist-test-slave-vrq1"
server_key: "34adf19beb442973c9845596f895dc41"
server_key_iv: "621b4406c77566e9e32d4c79f1680fad"
server_key_version: "encryptionkey@0"
I20260621 05:10:44.403311 9681 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260621 05:10:44.404145 9696 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 05:10:44.404373 9681 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260621 05:10:44.404461 9681 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782018616648578-3598-0/minicluster-data/master-0/data,/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782018616648578-3598-0/minicluster-data/master-0/wal
uuid: "08adad2fec7f4618bfbb0504365c5ecc"
format_stamp: "Formatted at 2026-06-21 05:10:44 on dist-test-slave-vrq1"
server_key: "34adf19beb442973c9845596f895dc41"
server_key_iv: "621b4406c77566e9e32d4c79f1680fad"
server_key_version: "encryptionkey@0"
I20260621 05:10:44.404551 9681 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782018616648578-3598-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782018616648578-3598-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782018616648578-3598-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 05:10:44.423539 9681 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 05:10:44.425976 9681 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 05:10:44.426214 9681 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 05:10:44.430900 9681 rpc_server.cc:307] RPC server started. Bound to: 127.3.131.190:33159
I20260621 05:10:44.430931 9748 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.131.190:33159 every 8 connection(s)
I20260621 05:10:44.431398 9681 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782018616648578-3598-0/minicluster-data/master-0/data/info.pb
I20260621 05:10:44.432492 9749 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 05:10:44.434840 9749 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 08adad2fec7f4618bfbb0504365c5ecc: Bootstrap starting.
I20260621 05:10:44.434885 3598 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWZOx2_/build/release/bin/kudu as pid 9681
I20260621 05:10:44.435017 3598 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782018616648578-3598-0/minicluster-data/master-0/wal/instance
I20260621 05:10:44.435294 3598 external_mini_cluster.cc:1428] Setting key 1e87dbb1c16e0359e3ae7fbcd2bff66b
I20260621 05:10:44.435680 9749 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 08adad2fec7f4618bfbb0504365c5ecc: Neither blocks nor log segments found. Creating new log.
I20260621 05:10:44.436039 9749 log.cc:824] T 00000000000000000000000000000000 P 08adad2fec7f4618bfbb0504365c5ecc: Log is configured to *not* fsync() on all Append() calls
I20260621 05:10:44.436821 9749 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 08adad2fec7f4618bfbb0504365c5ecc: No bootstrap required, opened a new log
I20260621 05:10:44.439128 9749 raft_consensus.cc:348] T 00000000000000000000000000000000 P 08adad2fec7f4618bfbb0504365c5ecc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "08adad2fec7f4618bfbb0504365c5ecc" member_type: VOTER last_known_addr { host: "127.3.131.190" port: 33159 } }
I20260621 05:10:44.439344 9749 raft_consensus.cc:374] T 00000000000000000000000000000000 P 08adad2fec7f4618bfbb0504365c5ecc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 05:10:44.439383 9749 raft_consensus.cc:729] T 00000000000000000000000000000000 P 08adad2fec7f4618bfbb0504365c5ecc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 08adad2fec7f4618bfbb0504365c5ecc, State: Initialized, Role: FOLLOWER
I20260621 05:10:44.439522 9749 consensus_queue.cc:260] T 00000000000000000000000000000000 P 08adad2fec7f4618bfbb0504365c5ecc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "08adad2fec7f4618bfbb0504365c5ecc" member_type: VOTER last_known_addr { host: "127.3.131.190" port: 33159 } }
I20260621 05:10:44.439618 9749 raft_consensus.cc:388] T 00000000000000000000000000000000 P 08adad2fec7f4618bfbb0504365c5ecc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260621 05:10:44.439648 9749 raft_consensus.cc:482] T 00000000000000000000000000000000 P 08adad2fec7f4618bfbb0504365c5ecc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260621 05:10:44.439685 9749 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 08adad2fec7f4618bfbb0504365c5ecc [term 0 FOLLOWER]: Advancing to term 1
I20260621 05:10:44.440342 9749 raft_consensus.cc:504] T 00000000000000000000000000000000 P 08adad2fec7f4618bfbb0504365c5ecc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "08adad2fec7f4618bfbb0504365c5ecc" member_type: VOTER last_known_addr { host: "127.3.131.190" port: 33159 } }
I20260621 05:10:44.440478 9749 leader_election.cc:302] T 00000000000000000000000000000000 P 08adad2fec7f4618bfbb0504365c5ecc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 08adad2fec7f4618bfbb0504365c5ecc; no voters:
I20260621 05:10:44.440677 9749 leader_election.cc:288] T 00000000000000000000000000000000 P 08adad2fec7f4618bfbb0504365c5ecc [CANDIDATE]: Term 1 election: Requested vote from peers
I20260621 05:10:44.440804 9754 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 08adad2fec7f4618bfbb0504365c5ecc [term 1 FOLLOWER]: Leader election won for term 1
I20260621 05:10:44.440960 9754 raft_consensus.cc:686] T 00000000000000000000000000000000 P 08adad2fec7f4618bfbb0504365c5ecc [term 1 LEADER]: Becoming Leader. State: Replica: 08adad2fec7f4618bfbb0504365c5ecc, State: Running, Role: LEADER
I20260621 05:10:44.441067 9754 consensus_queue.cc:237] T 00000000000000000000000000000000 P 08adad2fec7f4618bfbb0504365c5ecc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "08adad2fec7f4618bfbb0504365c5ecc" member_type: VOTER last_known_addr { host: "127.3.131.190" port: 33159 } }
I20260621 05:10:44.441475 9755 sys_catalog.cc:455] T 00000000000000000000000000000000 P 08adad2fec7f4618bfbb0504365c5ecc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "08adad2fec7f4618bfbb0504365c5ecc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "08adad2fec7f4618bfbb0504365c5ecc" member_type: VOTER last_known_addr { host: "127.3.131.190" port: 33159 } } }
I20260621 05:10:44.441480 9756 sys_catalog.cc:455] T 00000000000000000000000000000000 P 08adad2fec7f4618bfbb0504365c5ecc [sys.catalog]: SysCatalogTable state changed. Reason: New leader 08adad2fec7f4618bfbb0504365c5ecc. Latest consensus state: current_term: 1 leader_uuid: "08adad2fec7f4618bfbb0504365c5ecc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "08adad2fec7f4618bfbb0504365c5ecc" member_type: VOTER last_known_addr { host: "127.3.131.190" port: 33159 } } }
I20260621 05:10:44.441586 9755 sys_catalog.cc:458] T 00000000000000000000000000000000 P 08adad2fec7f4618bfbb0504365c5ecc [sys.catalog]: This master's current role is: LEADER
I20260621 05:10:44.441591 9756 sys_catalog.cc:458] T 00000000000000000000000000000000 P 08adad2fec7f4618bfbb0504365c5ecc [sys.catalog]: This master's current role is: LEADER
I20260621 05:10:44.442091 9749 sys_catalog.cc:564] T 00000000000000000000000000000000 P 08adad2fec7f4618bfbb0504365c5ecc [sys.catalog]: configured and running, proceeding with master startup.
I20260621 05:10:44.442231 9758 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260621 05:10:44.442764 9758 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Jun 21 05:10:44 dist-test-slave-vrq1 krb5kdc[9668](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782018644, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.3.131.190@KRBTEST.COM
I20260621 05:10:44.444931 9758 catalog_manager.cc:1347] Generated new cluster ID: 48235deea928424ca976ad57e26663f4
I20260621 05:10:44.445008 9758 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260621 05:10:44.448120 9752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:10:44.435963 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53558 (local address 127.3.131.190:33159)
0621 05:10:44.436197 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0621 05:10:44.436207 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0621 05:10:44.436236 (+ 29us) server_negotiation.cc:407] Connection header received
0621 05:10:44.436827 (+ 591us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:10:44.436835 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:10:44.436897 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:10:44.437064 (+ 167us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:10:44.437869 (+ 805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:10:44.438805 (+ 936us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:10:44.439896 (+ 1091us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:10:44.440337 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:10:44.440428 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:10:44.443509 (+ 3081us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:10:44.443513 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:10:44.443513 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:10:44.443528 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:10:44.446276 (+ 2748us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:10:44.446752 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:10:44.446756 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:10:44.446757 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:10:44.446876 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:10:44.447087 (+ 211us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:10:44.447087 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:10:44.447088 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:10:44.447460 (+ 372us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:10:44.447650 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0621 05:10:44.447738 (+ 88us) server_negotiation.cc:299] Negotiation successful
0621 05:10:44.447893 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":176,"thread_start_us":113,"threads_started":1}
I20260621 05:10:44.453831 9758 catalog_manager.cc:1370] Generated new certificate authority record
I20260621 05:10:44.454407 9758 catalog_manager.cc:1504] Loading token signing keys...
I20260621 05:10:44.459275 9758 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 08adad2fec7f4618bfbb0504365c5ecc: Generated new TSK 0
I20260621 05:10:44.459509 9758 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.3.131.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.3.131.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.3.131.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782018616648578-3598-0/krb5kdc/oryx_127.3.131.129.keytab.
Entry for principal oryx/127.3.131.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782018616648578-3598-0/krb5kdc/oryx_127.3.131.129.keytab.
I20260621 05:10:44.490770 3598 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWZOx2_/build/release/bin/kudu
/tmp/dist-test-taskWZOx2_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782018616648578-3598-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782018616648578-3598-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782018616648578-3598-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782018616648578-3598-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.131.129:0
--local_ip_for_outbound_sockets=127.3.131.129
--webserver_interface=127.3.131.129
--webserver_port=0
--tserver_master_addrs=127.3.131.190:33159
--keytab_file=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782018616648578-3598-0/krb5kdc/oryx_127.3.131.129.keytab
--principal=oryx/127.3.131.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.131.148:35089
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782018616648578-3598-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782018616648578-3598-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782018616648578-3598-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 05:10:44.584136 9775 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 05:10:44.584381 9775 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 05:10:44.584410 9775 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 05:10:44.586020 9775 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 05:10:44.586073 9775 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 05:10:44.586107 9775 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.131.129
I20260621 05:10:44.587987 9775 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.131.148:35089
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782018616648578-3598-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782018616648578-3598-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782018616648578-3598-0/krb5kdc/oryx_127.3.131.129.keytab
--principal=oryx/127.3.131.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.131.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782018616648578-3598-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.131.129
--webserver_port=0
--tserver_master_addrs=127.3.131.190:33159
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9775
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.131.129
--log_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782018616648578-3598-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 21 Jun 2026 04:16:16 UTC on ccycloud.quasar-xycaha.root.comops.site
build id 4457881
I20260621 05:10:44.588256 9775 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 05:10:44.588528 9775 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 05:10:44.594193 9783 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:10:44.594239 9781 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:10:44.594354 9775 server_base.cc:1034] running on GCE node
W20260621 05:10:44.594242 9780 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:10:44.594646 9775 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 05:10:44.594926 9775 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 05:10:44.596138 9775 hybrid_clock.cc:648] HybridClock initialized: now 1782018644596090 us; error 26 us; skew 500 ppm
Jun 21 05:10:44 dist-test-slave-vrq1 krb5kdc[9668](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782018644, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.131.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 05:10:44.598989 9775 init.cc:377] Logged in from keytab as oryx/127.3.131.129@KRBTEST.COM (short username oryx)
I20260621 05:10:44.599646 9775 webserver.cc:460] Webserver started at http://127.3.131.129:37023/ using document root <none> and password file <none>
I20260621 05:10:44.599915 9775 fs_manager.cc:359] Metadata directory not provided
I20260621 05:10:44.599990 9775 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 05:10:44.600157 9775 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 05:10:44.601321 9775 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782018616648578-3598-0/minicluster-data/ts-0/data/instance:
uuid: "e3bf3241a75e43fca924eac143beb63f"
format_stamp: "Formatted at 2026-06-21 05:10:44 on dist-test-slave-vrq1"
server_key: "a138dea9d5e4c03b21a7cc229054417a"
server_key_iv: "3772845953935a001ad243fcd48dbf99"
server_key_version: "encryptionkey@0"
I20260621 05:10:44.601722 9775 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782018616648578-3598-0/minicluster-data/ts-0/wal/instance:
uuid: "e3bf3241a75e43fca924eac143beb63f"
format_stamp: "Formatted at 2026-06-21 05:10:44 on dist-test-slave-vrq1"
server_key: "a138dea9d5e4c03b21a7cc229054417a"
server_key_iv: "3772845953935a001ad243fcd48dbf99"
server_key_version: "encryptionkey@0"
I20260621 05:10:44.603255 9775 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260621 05:10:44.604053 9790 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 05:10:44.604279 9775 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260621 05:10:44.604391 9775 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782018616648578-3598-0/minicluster-data/ts-0/data,/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782018616648578-3598-0/minicluster-data/ts-0/wal
uuid: "e3bf3241a75e43fca924eac143beb63f"
format_stamp: "Formatted at 2026-06-21 05:10:44 on dist-test-slave-vrq1"
server_key: "a138dea9d5e4c03b21a7cc229054417a"
server_key_iv: "3772845953935a001ad243fcd48dbf99"
server_key_version: "encryptionkey@0"
I20260621 05:10:44.604471 9775 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782018616648578-3598-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782018616648578-3598-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782018616648578-3598-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 05:10:44.636233 9775 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 05:10:44.638893 9775 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 05:10:44.639083 9775 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 05:10:44.639381 9775 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 05:10:44.639724 9775 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 05:10:44.639783 9775 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 05:10:44.639830 9775 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 05:10:44.639868 9775 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 05:10:44.647300 9775 rpc_server.cc:307] RPC server started. Bound to: 127.3.131.129:38671
I20260621 05:10:44.647341 9903 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.131.129:38671 every 8 connection(s)
I20260621 05:10:44.648123 9775 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782018616648578-3598-0/minicluster-data/ts-0/data/info.pb
I20260621 05:10:44.656911 3598 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWZOx2_/build/release/bin/kudu as pid 9775
I20260621 05:10:44.657042 3598 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782018616648578-3598-0/minicluster-data/ts-0/wal/instance
I20260621 05:10:44.657291 3598 external_mini_cluster.cc:1428] Setting key 8b12f483ffceea110b8de608ba7e6b50
Jun 21 05:10:44 dist-test-slave-vrq1 krb5kdc[9668](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782018644, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.131.129@KRBTEST.COM for oryx/127.3.131.190@KRBTEST.COM
I20260621 05:10:44.662206 9752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:10:44.649047 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.131.129:35135 (local address 127.3.131.190:33159)
0621 05:10:44.649099 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0621 05:10:44.649101 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:10:44.649411 (+ 310us) server_negotiation.cc:407] Connection header received
0621 05:10:44.650311 (+ 900us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:10:44.650312 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:10:44.650327 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:10:44.650387 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:10:44.652373 (+ 1986us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:10:44.653224 (+ 851us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:10:44.654905 (+ 1681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:10:44.655209 (+ 304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:10:44.655285 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:10:44.658401 (+ 3116us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:10:44.658403 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:10:44.658403 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:10:44.658412 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:10:44.660657 (+ 2245us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:10:44.661113 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:10:44.661114 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:10:44.661115 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:10:44.661178 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:10:44.661447 (+ 269us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:10:44.661448 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:10:44.661448 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:10:44.661600 (+ 152us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:10:44.661682 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0621 05:10:44.661992 (+ 310us) server_negotiation.cc:299] Negotiation successful
0621 05:10:44.662087 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260621 05:10:44.662436 9906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:10:44.649037 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.131.190:33159 (local address 127.3.131.129:35135)
0621 05:10:44.649228 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0621 05:10:44.649304 (+ 76us) client_negotiation.cc:174] Beginning negotiation
0621 05:10:44.650178 (+ 874us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:10:44.650523 (+ 345us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:10:44.650529 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:10:44.651903 (+ 1374us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:10:44.652237 (+ 334us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:10:44.652245 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:10:44.653373 (+ 1128us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:10:44.653375 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:10:44.654512 (+ 1137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:10:44.654517 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:10:44.655295 (+ 778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:10:44.655297 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:10:44.655445 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:10:44.656400 (+ 955us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 05:10:44.656411 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:10:44.658263 (+ 1852us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 05:10:44.660752 (+ 2489us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:10:44.660761 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:10:44.660763 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:10:44.661009 (+ 246us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:10:44.661250 (+ 241us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:10:44.661251 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:10:44.661252 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:10:44.661368 (+ 116us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:10:44.661671 (+ 303us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 05:10:44.661679 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 05:10:44.661853 (+ 174us) client_negotiation.cc:769] Sending connection context
0621 05:10:44.661972 (+ 119us) client_negotiation.cc:240] Negotiation successful
0621 05:10:44.662167 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":107,"threads_started":1}
I20260621 05:10:44.663075 9904 heartbeater.cc:347] Connected to a master server at 127.3.131.190:33159
I20260621 05:10:44.663226 9904 heartbeater.cc:464] Registering TS with master...
I20260621 05:10:44.663517 9904 heartbeater.cc:511] Master 127.3.131.190:33159 requested a full tablet report, sending...
I20260621 05:10:44.664201 9713 ts_manager.cc:194] Registered new tserver with Master: e3bf3241a75e43fca924eac143beb63f (127.3.131.129:38671)
I20260621 05:10:44.665155 9713 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.3.131.129@KRBTEST.COM'} at 127.3.131.129:35135
WARNING: no policy specified for oryx/127.3.131.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.3.131.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.3.131.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782018616648578-3598-0/krb5kdc/oryx_127.3.131.130.keytab.
Entry for principal oryx/127.3.131.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782018616648578-3598-0/krb5kdc/oryx_127.3.131.130.keytab.
I20260621 05:10:44.691195 3598 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWZOx2_/build/release/bin/kudu
/tmp/dist-test-taskWZOx2_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782018616648578-3598-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782018616648578-3598-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782018616648578-3598-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782018616648578-3598-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.131.130:0
--local_ip_for_outbound_sockets=127.3.131.130
--webserver_interface=127.3.131.130
--webserver_port=0
--tserver_master_addrs=127.3.131.190:33159
--keytab_file=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782018616648578-3598-0/krb5kdc/oryx_127.3.131.130.keytab
--principal=oryx/127.3.131.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.131.148:35089
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782018616648578-3598-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782018616648578-3598-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782018616648578-3598-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 05:10:44.781541 9909 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 05:10:44.781833 9909 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 05:10:44.781878 9909 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 05:10:44.783352 9909 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 05:10:44.783432 9909 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 05:10:44.783488 9909 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.131.130
I20260621 05:10:44.785416 9909 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.131.148:35089
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782018616648578-3598-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782018616648578-3598-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782018616648578-3598-0/krb5kdc/oryx_127.3.131.130.keytab
--principal=oryx/127.3.131.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.131.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782018616648578-3598-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.131.130
--webserver_port=0
--tserver_master_addrs=127.3.131.190:33159
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9909
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.131.130
--log_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782018616648578-3598-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 21 Jun 2026 04:16:16 UTC on ccycloud.quasar-xycaha.root.comops.site
build id 4457881
I20260621 05:10:44.785805 9909 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 05:10:44.786142 9909 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 05:10:44.792145 9914 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:10:44.792106 9915 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:10:44.792106 9917 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:10:44.792461 9909 server_base.cc:1034] running on GCE node
I20260621 05:10:44.792642 9909 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 05:10:44.792857 9909 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 05:10:44.794045 9909 hybrid_clock.cc:648] HybridClock initialized: now 1782018644794036 us; error 39 us; skew 500 ppm
Jun 21 05:10:44 dist-test-slave-vrq1 krb5kdc[9668](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782018644, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.131.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 05:10:44.796897 9909 init.cc:377] Logged in from keytab as oryx/127.3.131.130@KRBTEST.COM (short username oryx)
I20260621 05:10:44.797325 9909 webserver.cc:460] Webserver started at http://127.3.131.130:37661/ using document root <none> and password file <none>
I20260621 05:10:44.797560 9909 fs_manager.cc:359] Metadata directory not provided
I20260621 05:10:44.797612 9909 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 05:10:44.797745 9909 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 05:10:44.798815 9909 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782018616648578-3598-0/minicluster-data/ts-1/data/instance:
uuid: "8243bd6da6a04cdb80f9f0c681600639"
format_stamp: "Formatted at 2026-06-21 05:10:44 on dist-test-slave-vrq1"
server_key: "47d063eac3a7917bf4aa6b3561563013"
server_key_iv: "db8daa4d92d8eecc16660176762271b7"
server_key_version: "encryptionkey@0"
I20260621 05:10:44.799223 9909 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782018616648578-3598-0/minicluster-data/ts-1/wal/instance:
uuid: "8243bd6da6a04cdb80f9f0c681600639"
format_stamp: "Formatted at 2026-06-21 05:10:44 on dist-test-slave-vrq1"
server_key: "47d063eac3a7917bf4aa6b3561563013"
server_key_iv: "db8daa4d92d8eecc16660176762271b7"
server_key_version: "encryptionkey@0"
I20260621 05:10:44.800680 9909 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260621 05:10:44.801514 9924 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 05:10:44.801745 9909 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260621 05:10:44.801855 9909 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782018616648578-3598-0/minicluster-data/ts-1/data,/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782018616648578-3598-0/minicluster-data/ts-1/wal
uuid: "8243bd6da6a04cdb80f9f0c681600639"
format_stamp: "Formatted at 2026-06-21 05:10:44 on dist-test-slave-vrq1"
server_key: "47d063eac3a7917bf4aa6b3561563013"
server_key_iv: "db8daa4d92d8eecc16660176762271b7"
server_key_version: "encryptionkey@0"
I20260621 05:10:44.801942 9909 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782018616648578-3598-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782018616648578-3598-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782018616648578-3598-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 05:10:44.823722 9909 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 05:10:44.826357 9909 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 05:10:44.826651 9909 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 05:10:44.826924 9909 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 05:10:44.827386 9909 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 05:10:44.827442 9909 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 05:10:44.827484 9909 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 05:10:44.827513 9909 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 05:10:44.833616 9909 rpc_server.cc:307] RPC server started. Bound to: 127.3.131.130:34425
I20260621 05:10:44.833659 10037 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.131.130:34425 every 8 connection(s)
I20260621 05:10:44.833998 9909 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782018616648578-3598-0/minicluster-data/ts-1/data/info.pb
I20260621 05:10:44.837380 3598 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWZOx2_/build/release/bin/kudu as pid 9909
I20260621 05:10:44.837504 3598 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782018616648578-3598-0/minicluster-data/ts-1/wal/instance
I20260621 05:10:44.837721 3598 external_mini_cluster.cc:1428] Setting key 6dfa49c0e98dbb51de80411f4b7c1a39
Jun 21 05:10:44 dist-test-slave-vrq1 krb5kdc[9668](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782018644, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.131.130@KRBTEST.COM for oryx/127.3.131.190@KRBTEST.COM
I20260621 05:10:44.844981 9752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:10:44.834802 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.131.130:40399 (local address 127.3.131.190:33159)
0621 05:10:44.834885 (+ 83us) server_negotiation.cc:206] Beginning negotiation
0621 05:10:44.834888 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0621 05:10:44.835150 (+ 262us) server_negotiation.cc:407] Connection header received
0621 05:10:44.835931 (+ 781us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:10:44.835932 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:10:44.835942 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:10:44.835995 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:10:44.836912 (+ 917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:10:44.837411 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:10:44.838845 (+ 1434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:10:44.839060 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:10:44.839141 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:10:44.841753 (+ 2612us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:10:44.841754 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:10:44.841755 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:10:44.841763 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:10:44.843350 (+ 1587us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:10:44.843804 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:10:44.843805 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:10:44.843805 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:10:44.843869 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:10:44.844167 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:10:44.844169 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:10:44.844169 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:10:44.844350 (+ 181us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:10:44.844449 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0621 05:10:44.844764 (+ 315us) server_negotiation.cc:299] Negotiation successful
0621 05:10:44.844861 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260621 05:10:44.845106 10040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:10:44.834774 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.131.190:33159 (local address 127.3.131.130:40399)
0621 05:10:44.834979 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0621 05:10:44.835036 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0621 05:10:44.835844 (+ 808us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:10:44.836052 (+ 208us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:10:44.836055 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:10:44.836636 (+ 581us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:10:44.836829 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:10:44.836834 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:10:44.837515 (+ 681us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:10:44.837517 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:10:44.838752 (+ 1235us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:10:44.838754 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:10:44.839158 (+ 404us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:10:44.839160 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:10:44.839338 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:10:44.839938 (+ 600us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 05:10:44.839946 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:10:44.841617 (+ 1671us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 05:10:44.843440 (+ 1823us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:10:44.843449 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:10:44.843452 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:10:44.843718 (+ 266us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:10:44.843973 (+ 255us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:10:44.843974 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:10:44.843974 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:10:44.844093 (+ 119us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:10:44.844457 (+ 364us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 05:10:44.844467 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 05:10:44.844606 (+ 139us) client_negotiation.cc:769] Sending connection context
0621 05:10:44.844718 (+ 112us) client_negotiation.cc:240] Negotiation successful
0621 05:10:44.844873 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":151,"thread_start_us":100,"threads_started":1}
I20260621 05:10:44.845767 10038 heartbeater.cc:347] Connected to a master server at 127.3.131.190:33159
I20260621 05:10:44.845888 10038 heartbeater.cc:464] Registering TS with master...
I20260621 05:10:44.846073 10038 heartbeater.cc:511] Master 127.3.131.190:33159 requested a full tablet report, sending...
I20260621 05:10:44.846561 9713 ts_manager.cc:194] Registered new tserver with Master: 8243bd6da6a04cdb80f9f0c681600639 (127.3.131.130:34425)
WARNING: no policy specified for oryx/127.3.131.131@KRBTEST.COM; defaulting to no policy
I20260621 05:10:44.847092 9713 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.3.131.130@KRBTEST.COM'} at 127.3.131.130:40399
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.3.131.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.3.131.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782018616648578-3598-0/krb5kdc/oryx_127.3.131.131.keytab.
Entry for principal oryx/127.3.131.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782018616648578-3598-0/krb5kdc/oryx_127.3.131.131.keytab.
I20260621 05:10:44.867806 3598 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWZOx2_/build/release/bin/kudu
/tmp/dist-test-taskWZOx2_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782018616648578-3598-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782018616648578-3598-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782018616648578-3598-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782018616648578-3598-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.131.131:0
--local_ip_for_outbound_sockets=127.3.131.131
--webserver_interface=127.3.131.131
--webserver_port=0
--tserver_master_addrs=127.3.131.190:33159
--keytab_file=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782018616648578-3598-0/krb5kdc/oryx_127.3.131.131.keytab
--principal=oryx/127.3.131.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.131.148:35089
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782018616648578-3598-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782018616648578-3598-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782018616648578-3598-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 05:10:44.958070 10043 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 05:10:44.958317 10043 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 05:10:44.958348 10043 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 05:10:44.960042 10043 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 05:10:44.960105 10043 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 05:10:44.960140 10043 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.131.131
I20260621 05:10:44.961700 10043 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.131.148:35089
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782018616648578-3598-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782018616648578-3598-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782018616648578-3598-0/krb5kdc/oryx_127.3.131.131.keytab
--principal=oryx/127.3.131.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.131.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782018616648578-3598-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.131.131
--webserver_port=0
--tserver_master_addrs=127.3.131.190:33159
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10043
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.131.131
--log_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782018616648578-3598-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 21 Jun 2026 04:16:16 UTC on ccycloud.quasar-xycaha.root.comops.site
build id 4457881
I20260621 05:10:44.961913 10043 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 05:10:44.962101 10043 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 05:10:44.967475 10051 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:10:44.967495 10048 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:10:44.967475 10049 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:10:44.967852 10043 server_base.cc:1034] running on GCE node
I20260621 05:10:44.968056 10043 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 05:10:44.968324 10043 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 05:10:44.969496 10043 hybrid_clock.cc:648] HybridClock initialized: now 1782018644969494 us; error 41 us; skew 500 ppm
Jun 21 05:10:44 dist-test-slave-vrq1 krb5kdc[9668](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782018644, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.131.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 05:10:44.972222 10043 init.cc:377] Logged in from keytab as oryx/127.3.131.131@KRBTEST.COM (short username oryx)
I20260621 05:10:44.972697 10043 webserver.cc:460] Webserver started at http://127.3.131.131:44923/ using document root <none> and password file <none>
I20260621 05:10:44.972926 10043 fs_manager.cc:359] Metadata directory not provided
I20260621 05:10:44.972996 10043 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 05:10:44.973168 10043 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 05:10:44.974221 10043 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782018616648578-3598-0/minicluster-data/ts-2/data/instance:
uuid: "7e1eb017cbe947a08ad9b9e092651410"
format_stamp: "Formatted at 2026-06-21 05:10:44 on dist-test-slave-vrq1"
server_key: "65010e9c28dcf53ff9c5cf203e36540c"
server_key_iv: "5888a21b0180e69ce32f332b500ce118"
server_key_version: "encryptionkey@0"
I20260621 05:10:44.974521 10043 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782018616648578-3598-0/minicluster-data/ts-2/wal/instance:
uuid: "7e1eb017cbe947a08ad9b9e092651410"
format_stamp: "Formatted at 2026-06-21 05:10:44 on dist-test-slave-vrq1"
server_key: "65010e9c28dcf53ff9c5cf203e36540c"
server_key_iv: "5888a21b0180e69ce32f332b500ce118"
server_key_version: "encryptionkey@0"
I20260621 05:10:44.975910 10043 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260621 05:10:44.976583 10058 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 05:10:44.976765 10043 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260621 05:10:44.976835 10043 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782018616648578-3598-0/minicluster-data/ts-2/data,/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782018616648578-3598-0/minicluster-data/ts-2/wal
uuid: "7e1eb017cbe947a08ad9b9e092651410"
format_stamp: "Formatted at 2026-06-21 05:10:44 on dist-test-slave-vrq1"
server_key: "65010e9c28dcf53ff9c5cf203e36540c"
server_key_iv: "5888a21b0180e69ce32f332b500ce118"
server_key_version: "encryptionkey@0"
I20260621 05:10:44.976891 10043 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782018616648578-3598-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782018616648578-3598-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782018616648578-3598-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 05:10:44.996345 10043 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 05:10:44.998767 10043 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 05:10:44.998888 10043 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 05:10:44.999081 10043 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 05:10:44.999423 10043 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 05:10:44.999460 10043 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 05:10:44.999483 10043 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 05:10:44.999496 10043 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 05:10:45.005645 10043 rpc_server.cc:307] RPC server started. Bound to: 127.3.131.131:37567
I20260621 05:10:45.005731 10171 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.131.131:37567 every 8 connection(s)
I20260621 05:10:45.006047 10043 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782018616648578-3598-0/minicluster-data/ts-2/data/info.pb
Jun 21 05:10:45 dist-test-slave-vrq1 krb5kdc[9668](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782018644, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.131.131@KRBTEST.COM for oryx/127.3.131.190@KRBTEST.COM
I20260621 05:10:45.013890 3598 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWZOx2_/build/release/bin/kudu as pid 10043
I20260621 05:10:45.014019 3598 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782018616648578-3598-0/minicluster-data/ts-2/wal/instance
I20260621 05:10:45.014240 3598 external_mini_cluster.cc:1428] Setting key 4f2b24b602f6df15d3efe50a141c7e26
I20260621 05:10:45.016821 9752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:10:45.006811 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.131.131:39431 (local address 127.3.131.190:33159)
0621 05:10:45.006889 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0621 05:10:45.006892 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0621 05:10:45.007067 (+ 175us) server_negotiation.cc:407] Connection header received
0621 05:10:45.007730 (+ 663us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:10:45.007731 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:10:45.007744 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:10:45.007822 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:10:45.008856 (+ 1034us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:10:45.009360 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:10:45.010368 (+ 1008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:10:45.010599 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:10:45.010673 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:10:45.013162 (+ 2489us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:10:45.013164 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:10:45.013164 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:10:45.013174 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:10:45.015227 (+ 2053us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:10:45.015762 (+ 535us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:10:45.015763 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:10:45.015763 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:10:45.015831 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:10:45.016120 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:10:45.016121 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:10:45.016122 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:10:45.016256 (+ 134us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:10:45.016337 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0621 05:10:45.016631 (+ 294us) server_negotiation.cc:299] Negotiation successful
0621 05:10:45.016726 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260621 05:10:45.017059 10174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:10:45.006765 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.131.190:33159 (local address 127.3.131.131:39431)
0621 05:10:45.006932 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0621 05:10:45.006974 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0621 05:10:45.007620 (+ 646us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:10:45.007877 (+ 257us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:10:45.007881 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:10:45.008565 (+ 684us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:10:45.008772 (+ 207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:10:45.008778 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:10:45.009454 (+ 676us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:10:45.009454 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:10:45.010262 (+ 808us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:10:45.010265 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:10:45.010684 (+ 419us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:10:45.010684 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:10:45.010783 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:10:45.011410 (+ 627us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 05:10:45.011418 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:10:45.013029 (+ 1611us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 05:10:45.015369 (+ 2340us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:10:45.015377 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:10:45.015379 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:10:45.015629 (+ 250us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:10:45.015909 (+ 280us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:10:45.015909 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:10:45.015910 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:10:45.016032 (+ 122us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:10:45.016326 (+ 294us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 05:10:45.016333 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 05:10:45.016487 (+ 154us) client_negotiation.cc:769] Sending connection context
0621 05:10:45.016607 (+ 120us) client_negotiation.cc:240] Negotiation successful
0621 05:10:45.016781 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":92,"threads_started":1}
I20260621 05:10:45.017711 10172 heartbeater.cc:347] Connected to a master server at 127.3.131.190:33159
I20260621 05:10:45.017810 10172 heartbeater.cc:464] Registering TS with master...
I20260621 05:10:45.018071 10172 heartbeater.cc:511] Master 127.3.131.190:33159 requested a full tablet report, sending...
I20260621 05:10:45.018540 9713 ts_manager.cc:194] Registered new tserver with Master: 7e1eb017cbe947a08ad9b9e092651410 (127.3.131.131:37567)
I20260621 05:10:45.018889 9713 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.3.131.131@KRBTEST.COM'} at 127.3.131.131:39431
I20260621 05:10:45.026659 3598 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.3.131.189@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.3.131.189@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.3.131.189 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782018616648578-3598-0/krb5kdc/oryx_127.3.131.189.keytab.
Entry for principal oryx/127.3.131.189 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782018616648578-3598-0/krb5kdc/oryx_127.3.131.189.keytab.
I20260621 05:10:45.056378 3598 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWZOx2_/build/release/bin/kudu
/tmp/dist-test-taskWZOx2_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782018616648578-3598-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782018616648578-3598-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782018616648578-3598-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782018616648578-3598-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.131.189:41383
--webserver_interface=127.3.131.189
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782018616648578-3598-0/krb5kdc/oryx_127.3.131.189.keytab
--principal=oryx/127.3.131.189
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.131.148:35089
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.131.190:33159,127.3.131.189:41383
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782018616648578-3598-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782018616648578-3598-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782018616648578-3598-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 05:10:45.140959 10177 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 05:10:45.141194 10177 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 05:10:45.141222 10177 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 05:10:45.142438 10177 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 05:10:45.142483 10177 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 05:10:45.142498 10177 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 05:10:45.142511 10177 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 05:10:45.142524 10177 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 05:10:45.144232 10177 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.131.148:35089
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782018616648578-3598-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782018616648578-3598-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.3.131.190:33159,127.3.131.189:41383
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782018616648578-3598-0/krb5kdc/oryx_127.3.131.189.keytab
--principal=oryx/127.3.131.189
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.131.189:41383
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782018616648578-3598-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.3.131.189
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10177
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782018616648578-3598-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 21 Jun 2026 04:16:16 UTC on ccycloud.quasar-xycaha.root.comops.site
build id 4457881
W20260621 05:10:45.144351 10177 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.3.131.190:33159,127.3.131.189:41383'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260621 05:10:45.144474 10177 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 05:10:45.144649 10177 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 05:10:45.149732 10183 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:10:45.149893 10182 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:10:45.149976 10185 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:10:45.149945 10177 server_base.cc:1034] running on GCE node
I20260621 05:10:45.150291 10177 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 05:10:45.150611 10177 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 05:10:45.151764 10177 hybrid_clock.cc:648] HybridClock initialized: now 1782018645151745 us; error 56 us; skew 500 ppm
Jun 21 05:10:45 dist-test-slave-vrq1 krb5kdc[9668](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782018645, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.131.189@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 05:10:45.154500 10177 init.cc:377] Logged in from keytab as oryx/127.3.131.189@KRBTEST.COM (short username oryx)
I20260621 05:10:45.155031 10177 webserver.cc:460] Webserver started at http://127.3.131.189:45817/ using document root <none> and password file <none>
I20260621 05:10:45.155315 10177 fs_manager.cc:359] Metadata directory not provided
I20260621 05:10:45.155365 10177 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 05:10:45.155498 10177 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 05:10:45.156636 10177 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782018616648578-3598-0/minicluster-data/master-1/data/instance:
uuid: "65fa57b1227645fa8b95922a4da9e2ef"
format_stamp: "Formatted at 2026-06-21 05:10:45 on dist-test-slave-vrq1"
server_key: "b1f79c6711c4dd2227807b6daef93fac"
server_key_iv: "d18bc3d73a65da9404ddae335be02bc1"
server_key_version: "encryptionkey@0"
I20260621 05:10:45.157020 10177 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782018616648578-3598-0/minicluster-data/master-1/wal/instance:
uuid: "65fa57b1227645fa8b95922a4da9e2ef"
format_stamp: "Formatted at 2026-06-21 05:10:45 on dist-test-slave-vrq1"
server_key: "b1f79c6711c4dd2227807b6daef93fac"
server_key_iv: "d18bc3d73a65da9404ddae335be02bc1"
server_key_version: "encryptionkey@0"
I20260621 05:10:45.158432 10177 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260621 05:10:45.159309 10192 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 05:10:45.159569 10177 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260621 05:10:45.159662 10177 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782018616648578-3598-0/minicluster-data/master-1/data,/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782018616648578-3598-0/minicluster-data/master-1/wal
uuid: "65fa57b1227645fa8b95922a4da9e2ef"
format_stamp: "Formatted at 2026-06-21 05:10:45 on dist-test-slave-vrq1"
server_key: "b1f79c6711c4dd2227807b6daef93fac"
server_key_iv: "d18bc3d73a65da9404ddae335be02bc1"
server_key_version: "encryptionkey@0"
I20260621 05:10:45.159775 10177 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782018616648578-3598-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782018616648578-3598-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782018616648578-3598-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 05:10:45.184296 10177 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 05:10:45.187424 10177 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 05:10:45.187603 10177 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 05:10:45.192957 10177 rpc_server.cc:307] RPC server started. Bound to: 127.3.131.189:41383
I20260621 05:10:45.192986 10244 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.131.189:41383 every 8 connection(s)
I20260621 05:10:45.193405 10177 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782018616648578-3598-0/minicluster-data/master-1/data/info.pb
I20260621 05:10:45.194396 10245 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 05:10:45.196035 10245 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.131.190" port: 33159 } has no permanent_uuid. Determining permanent_uuid...
Jun 21 05:10:45 dist-test-slave-vrq1 krb5kdc[9668](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782018645, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.131.189@KRBTEST.COM for oryx/127.3.131.190@KRBTEST.COM
I20260621 05:10:45.202737 3598 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWZOx2_/build/release/bin/kudu as pid 10177
I20260621 05:10:45.206080 9752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:10:45.196815 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53570 (local address 127.3.131.190:33159)
0621 05:10:45.196912 (+ 97us) server_negotiation.cc:206] Beginning negotiation
0621 05:10:45.196915 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0621 05:10:45.197043 (+ 128us) server_negotiation.cc:407] Connection header received
0621 05:10:45.197545 (+ 502us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:10:45.197546 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:10:45.197556 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:10:45.197611 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:10:45.198183 (+ 572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:10:45.198661 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:10:45.199783 (+ 1122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:10:45.200027 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:10:45.200094 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:10:45.202347 (+ 2253us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:10:45.202349 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:10:45.202349 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:10:45.202356 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:10:45.204272 (+ 1916us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:10:45.204898 (+ 626us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:10:45.204899 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:10:45.204900 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:10:45.204973 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:10:45.205284 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:10:45.205285 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:10:45.205285 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:10:45.205437 (+ 152us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:10:45.205585 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0621 05:10:45.205899 (+ 314us) server_negotiation.cc:299] Negotiation successful
0621 05:10:45.205985 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260621 05:10:45.206226 10247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:10:45.196740 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.131.190:33159 (local address 127.0.0.1:53570)
0621 05:10:45.196912 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0621 05:10:45.196960 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0621 05:10:45.197463 (+ 503us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:10:45.197667 (+ 204us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:10:45.197670 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:10:45.197915 (+ 245us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:10:45.198115 (+ 200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:10:45.198119 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:10:45.198742 (+ 623us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:10:45.198743 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:10:45.199689 (+ 946us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:10:45.199692 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:10:45.200134 (+ 442us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:10:45.200135 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:10:45.200260 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:10:45.200867 (+ 607us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 05:10:45.200875 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:10:45.202235 (+ 1360us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 05:10:45.204413 (+ 2178us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:10:45.204419 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:10:45.204421 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:10:45.204795 (+ 374us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:10:45.205061 (+ 266us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:10:45.205061 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:10:45.205062 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:10:45.205201 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:10:45.205591 (+ 390us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 05:10:45.205596 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 05:10:45.205767 (+ 171us) client_negotiation.cc:769] Sending connection context
0621 05:10:45.205857 (+ 90us) client_negotiation.cc:240] Negotiation successful
0621 05:10:45.205983 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":70,"threads_started":1}
I20260621 05:10:45.206851 10245 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.131.189" port: 41383 } has no permanent_uuid. Determining permanent_uuid...
Jun 21 05:10:45 dist-test-slave-vrq1 krb5kdc[9668](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782018644, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.3.131.189@KRBTEST.COM
Jun 21 05:10:45 dist-test-slave-vrq1 krb5kdc[9668](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782018645, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.131.189@KRBTEST.COM for oryx/127.3.131.189@KRBTEST.COM
I20260621 05:10:45.213459 10249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:10:45.203360 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35156 (local address 127.3.131.189:41383)
0621 05:10:45.203543 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0621 05:10:45.203548 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 05:10:45.203566 (+ 18us) server_negotiation.cc:407] Connection header received
0621 05:10:45.203626 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:10:45.203628 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:10:45.203639 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:10:45.203701 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:10:45.204447 (+ 746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:10:45.205452 (+ 1005us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:10:45.206645 (+ 1193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:10:45.207177 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:10:45.207281 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:10:45.210141 (+ 2860us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:10:45.210147 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:10:45.210147 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:10:45.210153 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:10:45.212187 (+ 2034us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:10:45.212546 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:10:45.212549 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:10:45.212549 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:10:45.212612 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:10:45.212797 (+ 185us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:10:45.212798 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:10:45.212799 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:10:45.213004 (+ 205us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:10:45.213190 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0621 05:10:45.213246 (+ 56us) server_negotiation.cc:299] Negotiation successful
0621 05:10:45.213325 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":154,"thread_start_us":107,"threads_started":1}
I20260621 05:10:45.214732 10247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:10:45.207137 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.131.189:41383 (local address 127.0.0.1:35170)
0621 05:10:45.207218 (+ 81us) negotiation.cc:107] Waiting for socket to connect
0621 05:10:45.207228 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0621 05:10:45.207264 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:10:45.207496 (+ 232us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:10:45.207497 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:10:45.207683 (+ 186us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:10:45.207744 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:10:45.207746 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:10:45.208522 (+ 776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:10:45.208523 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:10:45.209090 (+ 567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:10:45.209093 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:10:45.209448 (+ 355us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:10:45.209450 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:10:45.209523 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:10:45.209994 (+ 471us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 05:10:45.210000 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:10:45.211241 (+ 1241us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 05:10:45.213910 (+ 2669us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:10:45.213911 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:10:45.213912 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:10:45.214121 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:10:45.214284 (+ 163us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:10:45.214285 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:10:45.214285 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:10:45.214340 (+ 55us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:10:45.214524 (+ 184us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 05:10:45.214525 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 05:10:45.214560 (+ 35us) client_negotiation.cc:769] Sending connection context
0621 05:10:45.214604 (+ 44us) client_negotiation.cc:240] Negotiation successful
0621 05:10:45.214675 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":65}
I20260621 05:10:45.214849 10250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:10:45.207235 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35170 (local address 127.3.131.189:41383)
0621 05:10:45.207376 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0621 05:10:45.207378 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:10:45.207383 (+ 5us) server_negotiation.cc:407] Connection header received
0621 05:10:45.207401 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:10:45.207402 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:10:45.207411 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:10:45.207478 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:10:45.207828 (+ 350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:10:45.208424 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:10:45.209174 (+ 750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:10:45.209373 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:10:45.209427 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:10:45.211324 (+ 1897us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:10:45.211326 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:10:45.211326 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:10:45.211333 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:10:45.213839 (+ 2506us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:10:45.214183 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:10:45.214184 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:10:45.214184 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:10:45.214245 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:10:45.214391 (+ 146us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:10:45.214392 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:10:45.214392 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:10:45.214481 (+ 89us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:10:45.214525 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0621 05:10:45.214620 (+ 95us) server_negotiation.cc:299] Negotiation successful
0621 05:10:45.214719 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":42,"server-negotiator.queue_time_us":123,"thread_start_us":87,"threads_started":1}
I20260621 05:10:45.216161 10245 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 65fa57b1227645fa8b95922a4da9e2ef: Bootstrap starting.
I20260621 05:10:45.216804 10245 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 65fa57b1227645fa8b95922a4da9e2ef: Neither blocks nor log segments found. Creating new log.
I20260621 05:10:45.217090 10245 log.cc:824] T 00000000000000000000000000000000 P 65fa57b1227645fa8b95922a4da9e2ef: Log is configured to *not* fsync() on all Append() calls
I20260621 05:10:45.217837 10245 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 65fa57b1227645fa8b95922a4da9e2ef: No bootstrap required, opened a new log
I20260621 05:10:45.219305 10245 raft_consensus.cc:348] T 00000000000000000000000000000000 P 65fa57b1227645fa8b95922a4da9e2ef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08adad2fec7f4618bfbb0504365c5ecc" member_type: VOTER last_known_addr { host: "127.3.131.190" port: 33159 } } peers { permanent_uuid: "65fa57b1227645fa8b95922a4da9e2ef" member_type: VOTER last_known_addr { host: "127.3.131.189" port: 41383 } }
I20260621 05:10:45.219452 10245 raft_consensus.cc:374] T 00000000000000000000000000000000 P 65fa57b1227645fa8b95922a4da9e2ef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 05:10:45.219499 10245 raft_consensus.cc:729] T 00000000000000000000000000000000 P 65fa57b1227645fa8b95922a4da9e2ef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 65fa57b1227645fa8b95922a4da9e2ef, State: Initialized, Role: FOLLOWER
I20260621 05:10:45.219683 10245 consensus_queue.cc:260] T 00000000000000000000000000000000 P 65fa57b1227645fa8b95922a4da9e2ef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08adad2fec7f4618bfbb0504365c5ecc" member_type: VOTER last_known_addr { host: "127.3.131.190" port: 33159 } } peers { permanent_uuid: "65fa57b1227645fa8b95922a4da9e2ef" member_type: VOTER last_known_addr { host: "127.3.131.189" port: 41383 } }
I20260621 05:10:45.219961 10252 sys_catalog.cc:455] T 00000000000000000000000000000000 P 65fa57b1227645fa8b95922a4da9e2ef [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08adad2fec7f4618bfbb0504365c5ecc" member_type: VOTER last_known_addr { host: "127.3.131.190" port: 33159 } } peers { permanent_uuid: "65fa57b1227645fa8b95922a4da9e2ef" member_type: VOTER last_known_addr { host: "127.3.131.189" port: 41383 } } }
I20260621 05:10:45.220005 10245 sys_catalog.cc:564] T 00000000000000000000000000000000 P 65fa57b1227645fa8b95922a4da9e2ef [sys.catalog]: configured and running, proceeding with master startup.
I20260621 05:10:45.220050 10252 sys_catalog.cc:458] T 00000000000000000000000000000000 P 65fa57b1227645fa8b95922a4da9e2ef [sys.catalog]: This master's current role is: FOLLOWER
W20260621 05:10:45.221758 10263 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 65fa57b1227645fa8b95922a4da9e2ef: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260621 05:10:45.221820 10263 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260621 05:10:45.223604 10177 master_runner.cc:428] Detected that this master 65fa57b1227645fa8b95922a4da9e2ef is joining an existing cluster
I20260621 05:10:45.223654 10177 master_runner.cc:432] Initiating AddMaster RPC to add 127.3.131.189:41383
I20260621 05:10:45.231933 9752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:10:45.224734 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53584 (local address 127.3.131.190:33159)
0621 05:10:45.224785 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0621 05:10:45.224787 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:10:45.224961 (+ 174us) server_negotiation.cc:407] Connection header received
0621 05:10:45.224988 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:10:45.224989 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:10:45.225000 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:10:45.225086 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:10:45.225514 (+ 428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:10:45.226313 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:10:45.227438 (+ 1125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:10:45.227697 (+ 259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:10:45.227781 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:10:45.228916 (+ 1135us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:10:45.228918 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:10:45.228919 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:10:45.228935 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:10:45.230571 (+ 1636us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:10:45.231030 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:10:45.231031 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:10:45.231031 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:10:45.231086 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:10:45.231373 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:10:45.231374 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:10:45.231374 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:10:45.231496 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:10:45.231575 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0621 05:10:45.231751 (+ 176us) server_negotiation.cc:299] Negotiation successful
0621 05:10:45.231832 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260621 05:10:45.231948 10269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:10:45.224635 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.131.190:33159 (local address 127.0.0.1:53584)
0621 05:10:45.224788 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0621 05:10:45.224802 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0621 05:10:45.224871 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:10:45.225137 (+ 266us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:10:45.225138 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:10:45.225365 (+ 227us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:10:45.225431 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:10:45.225434 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:10:45.226396 (+ 962us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:10:45.226397 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:10:45.227341 (+ 944us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:10:45.227345 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:10:45.227778 (+ 433us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:10:45.227779 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:10:45.227846 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:10:45.228339 (+ 493us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 05:10:45.228344 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:10:45.228825 (+ 481us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 05:10:45.230668 (+ 1843us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:10:45.230669 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:10:45.230669 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:10:45.230949 (+ 280us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:10:45.231222 (+ 273us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:10:45.231223 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:10:45.231223 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:10:45.231295 (+ 72us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:10:45.231565 (+ 270us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 05:10:45.231566 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 05:10:45.231636 (+ 70us) client_negotiation.cc:769] Sending connection context
0621 05:10:45.231722 (+ 86us) client_negotiation.cc:240] Negotiation successful
0621 05:10:45.231832 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":98,"threads_started":1}
I20260621 05:10:45.233495 10270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:10:45.224880 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.131.189:41383 (local address 127.0.0.1:35172)
0621 05:10:45.224980 (+ 100us) negotiation.cc:107] Waiting for socket to connect
0621 05:10:45.224989 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0621 05:10:45.225034 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:10:45.225158 (+ 124us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:10:45.225159 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:10:45.225407 (+ 248us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:10:45.225473 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:10:45.225475 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:10:45.227445 (+ 1970us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:10:45.227445 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:10:45.228314 (+ 869us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:10:45.228317 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:10:45.228842 (+ 525us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:10:45.228843 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:10:45.228913 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:10:45.229331 (+ 418us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 05:10:45.229336 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:10:45.229709 (+ 373us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 05:10:45.231824 (+ 2115us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:10:45.231825 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:10:45.231825 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:10:45.232572 (+ 747us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:10:45.232781 (+ 209us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:10:45.232782 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:10:45.232782 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:10:45.232844 (+ 62us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:10:45.233155 (+ 311us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 05:10:45.233156 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 05:10:45.233193 (+ 37us) client_negotiation.cc:769] Sending connection context
0621 05:10:45.233262 (+ 69us) client_negotiation.cc:240] Negotiation successful
0621 05:10:45.233381 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":87,"thread_start_us":67,"threads_started":1}
I20260621 05:10:45.233920 10250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:10:45.224905 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35172 (local address 127.3.131.189:41383)
0621 05:10:45.224942 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0621 05:10:45.224944 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:10:45.225045 (+ 101us) server_negotiation.cc:407] Connection header received
0621 05:10:45.225073 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:10:45.225074 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:10:45.225083 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:10:45.225145 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:10:45.225571 (+ 426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:10:45.226303 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:10:45.228397 (+ 2094us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:10:45.228751 (+ 354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:10:45.228826 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:10:45.229803 (+ 977us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:10:45.229804 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:10:45.229805 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:10:45.229810 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:10:45.231740 (+ 1930us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:10:45.232658 (+ 918us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:10:45.232659 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:10:45.232659 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:10:45.232719 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:10:45.232919 (+ 200us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:10:45.232920 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:10:45.232921 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:10:45.233057 (+ 136us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:10:45.233143 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0621 05:10:45.233294 (+ 151us) server_negotiation.cc:299] Negotiation successful
0621 05:10:45.233829 (+ 535us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":200,"server-negotiator.queue_time_us":20}
Jun 21 05:10:45 dist-test-slave-vrq1 krb5kdc[9668](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782018644, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.131.190@KRBTEST.COM for oryx/127.3.131.189@KRBTEST.COM
I20260621 05:10:45.243103 10271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:10:45.234287 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.131.189:41383 (local address 127.0.0.1:35176)
0621 05:10:45.234549 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0621 05:10:45.234598 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0621 05:10:45.234678 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:10:45.234821 (+ 143us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:10:45.234823 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:10:45.235078 (+ 255us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:10:45.235134 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:10:45.235135 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:10:45.236143 (+ 1008us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:10:45.236144 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:10:45.237172 (+ 1028us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:10:45.237176 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:10:45.237583 (+ 407us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:10:45.237584 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:10:45.237679 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:10:45.238297 (+ 618us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 05:10:45.238303 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:10:45.240082 (+ 1779us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 05:10:45.241788 (+ 1706us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:10:45.241798 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:10:45.241800 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:10:45.242090 (+ 290us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:10:45.242317 (+ 227us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:10:45.242318 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:10:45.242318 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:10:45.242389 (+ 71us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:10:45.242664 (+ 275us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 05:10:45.242673 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 05:10:45.242737 (+ 64us) client_negotiation.cc:769] Sending connection context
0621 05:10:45.242829 (+ 92us) client_negotiation.cc:240] Negotiation successful
0621 05:10:45.242976 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":235,"thread_start_us":76,"threads_started":1}
I20260621 05:10:45.243108 10250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:10:45.234335 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35176 (local address 127.3.131.189:41383)
0621 05:10:45.234369 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0621 05:10:45.234371 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:10:45.234682 (+ 311us) server_negotiation.cc:407] Connection header received
0621 05:10:45.234737 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:10:45.234737 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:10:45.234748 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:10:45.234818 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:10:45.235262 (+ 444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:10:45.236025 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:10:45.237260 (+ 1235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:10:45.237471 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:10:45.237564 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:10:45.240207 (+ 2643us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:10:45.240209 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:10:45.240209 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:10:45.240217 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:10:45.241674 (+ 1457us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:10:45.242178 (+ 504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:10:45.242180 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:10:45.242180 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:10:45.242245 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:10:45.242454 (+ 209us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:10:45.242455 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:10:45.242455 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:10:45.242579 (+ 124us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:10:45.242651 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0621 05:10:45.242902 (+ 251us) server_negotiation.cc:299] Negotiation successful
0621 05:10:45.243021 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260621 05:10:45.243746 9713 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.3.131.189:41383: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "65fa57b1227645fa8b95922a4da9e2ef"
member_type: NON_VOTER
last_known_addr {
host: "127.3.131.189"
port: 41383
}
attrs {
promote: true
}
}
dest_uuid: "08adad2fec7f4618bfbb0504365c5ecc"
cas_config_opid_index: -1
I20260621 05:10:45.244120 9713 consensus_queue.cc:237] T 00000000000000000000000000000000 P 08adad2fec7f4618bfbb0504365c5ecc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "08adad2fec7f4618bfbb0504365c5ecc" member_type: VOTER last_known_addr { host: "127.3.131.190" port: 33159 } } peers { permanent_uuid: "65fa57b1227645fa8b95922a4da9e2ef" member_type: NON_VOTER last_known_addr { host: "127.3.131.189" port: 41383 } attrs { promote: true } }
I20260621 05:10:45.244621 10273 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 08adad2fec7f4618bfbb0504365c5ecc [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 65fa57b1227645fa8b95922a4da9e2ef (127.3.131.189) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "08adad2fec7f4618bfbb0504365c5ecc" member_type: VOTER last_known_addr { host: "127.3.131.190" port: 33159 } } peers { permanent_uuid: "65fa57b1227645fa8b95922a4da9e2ef" member_type: NON_VOTER last_known_addr { host: "127.3.131.189" port: 41383 } attrs { promote: true } } }
I20260621 05:10:45.245054 10219 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 65fa57b1227645fa8b95922a4da9e2ef [term 0 FOLLOWER]: Advancing to term 1
I20260621 05:10:45.245640 10273 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.3.131.189:41383
I20260621 05:10:45.245736 10273 sys_catalog.cc:455] T 00000000000000000000000000000000 P 08adad2fec7f4618bfbb0504365c5ecc [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "08adad2fec7f4618bfbb0504365c5ecc" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "08adad2fec7f4618bfbb0504365c5ecc" member_type: VOTER last_known_addr { host: "127.3.131.190" port: 33159 } } peers { permanent_uuid: "65fa57b1227645fa8b95922a4da9e2ef" member_type: NON_VOTER last_known_addr { host: "127.3.131.189" port: 41383 } attrs { promote: true } } }
I20260621 05:10:45.245805 10273 sys_catalog.cc:458] T 00000000000000000000000000000000 P 08adad2fec7f4618bfbb0504365c5ecc [sys.catalog]: This master's current role is: LEADER
I20260621 05:10:45.245905 10177 master.cc:537] Master@127.3.131.189:41383 shutting down...
I20260621 05:10:45.246027 10219 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 65fa57b1227645fa8b95922a4da9e2ef [term 1 FOLLOWER]: Refusing update from remote peer 08adad2fec7f4618bfbb0504365c5ecc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260621 05:10:45.246549 10272 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 08adad2fec7f4618bfbb0504365c5ecc [LEADER]: Connected to new peer: Peer: permanent_uuid: "65fa57b1227645fa8b95922a4da9e2ef" member_type: NON_VOTER last_known_addr { host: "127.3.131.189" port: 41383 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
W20260621 05:10:45.247570 9698 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
W20260621 05:10:45.248463 9698 consensus_peers.cc:487] T 00000000000000000000000000000000 P 08adad2fec7f4618bfbb0504365c5ecc -> Peer 65fa57b1227645fa8b95922a4da9e2ef (127.3.131.189:41383): Couldn't send request to peer 65fa57b1227645fa8b95922a4da9e2ef. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260621 05:10:45.248952 10177 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 65fa57b1227645fa8b95922a4da9e2ef [term 1 FOLLOWER]: Raft consensus shutting down.
I20260621 05:10:45.249050 10177 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 65fa57b1227645fa8b95922a4da9e2ef [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 05:10:45.249123 10177 tablet_replica.cc:330] T 00000000000000000000000000000000 P 65fa57b1227645fa8b95922a4da9e2ef: stopping tablet replica
I20260621 05:10:45.262594 10177 master.cc:552] Master@127.3.131.189:41383 shutdown complete.
I20260621 05:10:45.262653 10177 master_runner.cc:305] Clearing existing system tablet
I20260621 05:10:45.262811 10177 fs_manager.cc:359] Metadata directory not provided
I20260621 05:10:45.262871 10177 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 05:10:45.263677 10177 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.002s
I20260621 05:10:45.264209 10277 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 05:10:45.264377 10177 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782018616648578-3598-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 05:10:45.264432 10177 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260621 05:10:45.264456 10177 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782018616648578-3598-0/minicluster-data/master-1/data,/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782018616648578-3598-0/minicluster-data/master-1/wal
uuid: "65fa57b1227645fa8b95922a4da9e2ef"
format_stamp: "Formatted at 2026-06-21 05:10:45 on dist-test-slave-vrq1"
server_key: "b1f79c6711c4dd2227807b6daef93fac"
server_key_iv: "d18bc3d73a65da9404ddae335be02bc1"
server_key_version: "encryptionkey@0"
I20260621 05:10:45.265275 10177 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 65fa57b1227645fa8b95922a4da9e2ef: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 05:10:45.267360 10177 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 65fa57b1227645fa8b95922a4da9e2ef: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260621 05:10:45.267434 10177 log.cc:1196] T 00000000000000000000000000000000 P 65fa57b1227645fa8b95922a4da9e2ef: Deleting WAL directory at /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782018616648578-3598-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260621 05:10:45.267747 10177 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 65fa57b1227645fa8b95922a4da9e2ef: Deleting consensus metadata
I20260621 05:10:45.267897 10177 master_runner.cc:315] Copying system tablet from 127.3.131.190:33159
I20260621 05:10:45.268985 10177 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 65fa57b1227645fa8b95922a4da9e2ef: tablet copy: Beginning tablet copy session from remote peer at address 127.3.131.190:33159
I20260621 05:10:45.275745 9752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:10:45.269449 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53590 (local address 127.3.131.190:33159)
0621 05:10:45.269504 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0621 05:10:45.269506 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:10:45.269575 (+ 69us) server_negotiation.cc:407] Connection header received
0621 05:10:45.269728 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:10:45.269729 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:10:45.269741 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:10:45.269815 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:10:45.270167 (+ 352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:10:45.270701 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:10:45.271566 (+ 865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:10:45.271776 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:10:45.271865 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:10:45.272926 (+ 1061us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:10:45.272927 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:10:45.272928 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:10:45.272936 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:10:45.274417 (+ 1481us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:10:45.274796 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:10:45.274798 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:10:45.274798 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:10:45.274860 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:10:45.275060 (+ 200us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:10:45.275061 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:10:45.275061 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:10:45.275290 (+ 229us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:10:45.275376 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0621 05:10:45.275542 (+ 166us) server_negotiation.cc:299] Negotiation successful
0621 05:10:45.275626 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260621 05:10:45.275758 10283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:10:45.269373 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.131.190:33159 (local address 127.0.0.1:53590)
0621 05:10:45.269506 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0621 05:10:45.269518 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0621 05:10:45.269595 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:10:45.269820 (+ 225us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:10:45.269821 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:10:45.270014 (+ 193us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:10:45.270073 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:10:45.270075 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:10:45.270805 (+ 730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:10:45.270806 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:10:45.271474 (+ 668us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:10:45.271477 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:10:45.271867 (+ 390us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:10:45.271868 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:10:45.271944 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:10:45.272462 (+ 518us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 05:10:45.272466 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:10:45.272807 (+ 341us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 05:10:45.274521 (+ 1714us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:10:45.274522 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:10:45.274522 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:10:45.274712 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:10:45.274927 (+ 215us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:10:45.274928 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:10:45.274928 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:10:45.274995 (+ 67us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:10:45.275378 (+ 383us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 05:10:45.275379 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 05:10:45.275429 (+ 50us) client_negotiation.cc:769] Sending connection context
0621 05:10:45.275516 (+ 87us) client_negotiation.cc:240] Negotiation successful
0621 05:10:45.275628 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":72,"threads_started":1}
I20260621 05:10:45.276316 9733 tablet_copy_service.cc:140] P 08adad2fec7f4618bfbb0504365c5ecc: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 65fa57b1227645fa8b95922a4da9e2ef ({username='oryx', principal='oryx/127.3.131.189@KRBTEST.COM'} at 127.0.0.1:53590)
I20260621 05:10:45.276410 9733 tablet_copy_service.cc:161] P 08adad2fec7f4618bfbb0504365c5ecc: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 65fa57b1227645fa8b95922a4da9e2ef at {username='oryx', principal='oryx/127.3.131.189@KRBTEST.COM'} at 127.0.0.1:53590: session id = 65fa57b1227645fa8b95922a4da9e2ef-00000000000000000000000000000000
I20260621 05:10:45.277088 9733 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 08adad2fec7f4618bfbb0504365c5ecc: Tablet Copy: opened 0 blocks and 1 log segments
I20260621 05:10:45.277825 10177 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260621 05:10:45.278904 10177 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 65fa57b1227645fa8b95922a4da9e2ef: tablet copy: Starting download of 0 data blocks...
I20260621 05:10:45.279062 10177 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 65fa57b1227645fa8b95922a4da9e2ef: tablet copy: Starting download of 1 WAL segments...
I20260621 05:10:45.280030 10177 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 65fa57b1227645fa8b95922a4da9e2ef: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260621 05:10:45.280962 9733 tablet_copy_service.cc:342] P 08adad2fec7f4618bfbb0504365c5ecc: Request end of tablet copy session 65fa57b1227645fa8b95922a4da9e2ef-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.3.131.189@KRBTEST.COM'} at 127.0.0.1:53590
I20260621 05:10:45.281014 9733 tablet_copy_service.cc:433] P 08adad2fec7f4618bfbb0504365c5ecc: ending tablet copy session 65fa57b1227645fa8b95922a4da9e2ef-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 65fa57b1227645fa8b95922a4da9e2ef
I20260621 05:10:45.283959 10177 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 05:10:45.285267 10284 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:10:45.285328 10287 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:10:45.285287 10285 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:10:45.285660 10177 server_base.cc:1034] running on GCE node
I20260621 05:10:45.285758 10177 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 05:10:45.285916 10177 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 05:10:45.287084 10177 hybrid_clock.cc:648] HybridClock initialized: now 1782018645287075 us; error 25 us; skew 500 ppm
Jun 21 05:10:45 dist-test-slave-vrq1 krb5kdc[9668](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782018645, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.131.189@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 05:10:45.288659 10177 init.cc:377] Logged in from keytab as oryx/127.3.131.189@KRBTEST.COM (short username oryx)
I20260621 05:10:45.288976 10177 webserver.cc:460] Webserver started at http://127.3.131.189:44329/ using document root <none> and password file <none>
I20260621 05:10:45.289112 10177 fs_manager.cc:359] Metadata directory not provided
I20260621 05:10:45.289153 10177 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 05:10:45.289848 10177 fs_manager.cc:709] Time spent opening directory manager: real 0.000s user 0.000s sys 0.001s
I20260621 05:10:45.290438 10294 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 05:10:45.290577 10177 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260621 05:10:45.290627 10177 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782018616648578-3598-0/minicluster-data/master-1/data,/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782018616648578-3598-0/minicluster-data/master-1/wal
uuid: "65fa57b1227645fa8b95922a4da9e2ef"
format_stamp: "Formatted at 2026-06-21 05:10:45 on dist-test-slave-vrq1"
server_key: "b1f79c6711c4dd2227807b6daef93fac"
server_key_iv: "d18bc3d73a65da9404ddae335be02bc1"
server_key_version: "encryptionkey@0"
I20260621 05:10:45.290695 10177 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782018616648578-3598-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782018616648578-3598-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782018616648578-3598-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 05:10:45.302325 10177 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 05:10:45.304190 10177 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 05:10:45.307644 10177 rpc_server.cc:307] RPC server started. Bound to: 127.3.131.189:41383
I20260621 05:10:45.307657 10346 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.131.189:41383 every 8 connection(s)
I20260621 05:10:45.308060 10177 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782018616648578-3598-0/minicluster-data/master-1/data/info.pb
I20260621 05:10:45.309101 10347 sys_catalog.cc:263] Verifying existing consensus state
I20260621 05:10:45.309710 10347 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 65fa57b1227645fa8b95922a4da9e2ef: Bootstrap starting.
I20260621 05:10:45.312889 10347 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 65fa57b1227645fa8b95922a4da9e2ef: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260621 05:10:45.313145 10347 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 65fa57b1227645fa8b95922a4da9e2ef: Bootstrap complete.
I20260621 05:10:45.313470 10347 raft_consensus.cc:348] T 00000000000000000000000000000000 P 65fa57b1227645fa8b95922a4da9e2ef [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "08adad2fec7f4618bfbb0504365c5ecc" member_type: VOTER last_known_addr { host: "127.3.131.190" port: 33159 } } peers { permanent_uuid: "65fa57b1227645fa8b95922a4da9e2ef" member_type: NON_VOTER last_known_addr { host: "127.3.131.189" port: 41383 } attrs { promote: true } }
I20260621 05:10:45.313571 10347 raft_consensus.cc:729] T 00000000000000000000000000000000 P 65fa57b1227645fa8b95922a4da9e2ef [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 65fa57b1227645fa8b95922a4da9e2ef, State: Initialized, Role: LEARNER
I20260621 05:10:45.313613 10347 consensus_queue.cc:260] T 00000000000000000000000000000000 P 65fa57b1227645fa8b95922a4da9e2ef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "08adad2fec7f4618bfbb0504365c5ecc" member_type: VOTER last_known_addr { host: "127.3.131.190" port: 33159 } } peers { permanent_uuid: "65fa57b1227645fa8b95922a4da9e2ef" member_type: NON_VOTER last_known_addr { host: "127.3.131.189" port: 41383 } attrs { promote: true } }
I20260621 05:10:45.313807 10347 sys_catalog.cc:564] T 00000000000000000000000000000000 P 65fa57b1227645fa8b95922a4da9e2ef [sys.catalog]: configured and running, proceeding with master startup.
I20260621 05:10:45.313819 10350 sys_catalog.cc:455] T 00000000000000000000000000000000 P 65fa57b1227645fa8b95922a4da9e2ef [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "08adad2fec7f4618bfbb0504365c5ecc" member_type: VOTER last_known_addr { host: "127.3.131.190" port: 33159 } } peers { permanent_uuid: "65fa57b1227645fa8b95922a4da9e2ef" member_type: NON_VOTER last_known_addr { host: "127.3.131.189" port: 41383 } attrs { promote: true } } }
I20260621 05:10:45.313984 10350 sys_catalog.cc:458] T 00000000000000000000000000000000 P 65fa57b1227645fa8b95922a4da9e2ef [sys.catalog]: This master's current role is: LEARNER
I20260621 05:10:45.315407 10361 catalog_manager.cc:1259] Loaded cluster ID: 48235deea928424ca976ad57e26663f4
I20260621 05:10:45.315513 10361 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 65fa57b1227645fa8b95922a4da9e2ef: loading cluster ID for follower catalog manager: success
I20260621 05:10:45.316586 10361 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 65fa57b1227645fa8b95922a4da9e2ef: acquiring CA information for follower catalog manager: success
I20260621 05:10:45.316896 10361 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 65fa57b1227645fa8b95922a4da9e2ef: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260621 05:10:45.323750 10362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:10:45.315417 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35200 (local address 127.3.131.189:41383)
0621 05:10:45.317022 (+ 1605us) server_negotiation.cc:206] Beginning negotiation
0621 05:10:45.317024 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:10:45.317042 (+ 18us) server_negotiation.cc:407] Connection header received
0621 05:10:45.317069 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:10:45.317070 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:10:45.317083 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:10:45.317192 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:10:45.317762 (+ 570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:10:45.318411 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:10:45.319142 (+ 731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:10:45.319410 (+ 268us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:10:45.319487 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:10:45.320764 (+ 1277us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:10:45.320765 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:10:45.320766 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:10:45.320773 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:10:45.322392 (+ 1619us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:10:45.322846 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:10:45.322848 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:10:45.322848 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:10:45.322917 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:10:45.323122 (+ 205us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:10:45.323123 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:10:45.323124 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:10:45.323291 (+ 167us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:10:45.323419 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0621 05:10:45.323528 (+ 109us) server_negotiation.cc:299] Negotiation successful
0621 05:10:45.323621 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1558,"thread_start_us":95,"threads_started":1}
WARNING: no policy specified for oryx/127.3.131.188@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.3.131.188@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.3.131.188 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782018616648578-3598-0/krb5kdc/oryx_127.3.131.188.keytab.
Entry for principal oryx/127.3.131.188 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782018616648578-3598-0/krb5kdc/oryx_127.3.131.188.keytab.
I20260621 05:10:45.351559 3598 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWZOx2_/build/release/bin/kudu
/tmp/dist-test-taskWZOx2_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782018616648578-3598-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782018616648578-3598-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782018616648578-3598-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782018616648578-3598-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.131.188:32887
--webserver_interface=127.3.131.188
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782018616648578-3598-0/krb5kdc/oryx_127.3.131.188.keytab
--principal=oryx/127.3.131.188
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.131.148:35089
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.131.190:33159,127.3.131.189:41383,127.3.131.188:32887
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782018616648578-3598-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782018616648578-3598-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782018616648578-3598-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 05:10:45.443954 10365 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 05:10:45.444236 10365 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 05:10:45.444264 10365 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 05:10:45.445604 10365 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 05:10:45.445653 10365 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 05:10:45.445670 10365 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 05:10:45.445684 10365 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 05:10:45.445698 10365 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 05:10:45.447598 10365 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.131.148:35089
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782018616648578-3598-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782018616648578-3598-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.3.131.190:33159,127.3.131.189:41383,127.3.131.188:32887
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782018616648578-3598-0/krb5kdc/oryx_127.3.131.188.keytab
--principal=oryx/127.3.131.188
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.131.188:32887
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782018616648578-3598-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.3.131.188
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10365
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782018616648578-3598-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 21 Jun 2026 04:16:16 UTC on ccycloud.quasar-xycaha.root.comops.site
build id 4457881
I20260621 05:10:45.447901 10365 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 05:10:45.448173 10365 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 05:10:45.453270 10371 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:10:45.453270 10373 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:10:45.453320 10370 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:10:45.453647 10365 server_base.cc:1034] running on GCE node
I20260621 05:10:45.453900 10365 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 05:10:45.454267 10365 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 05:10:45.455420 10365 hybrid_clock.cc:648] HybridClock initialized: now 1782018645455396 us; error 39 us; skew 500 ppm
Jun 21 05:10:45 dist-test-slave-vrq1 krb5kdc[9668](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782018645, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.131.188@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 05:10:45.458199 10365 init.cc:377] Logged in from keytab as oryx/127.3.131.188@KRBTEST.COM (short username oryx)
I20260621 05:10:45.458725 10365 webserver.cc:460] Webserver started at http://127.3.131.188:33063/ using document root <none> and password file <none>
I20260621 05:10:45.458976 10365 fs_manager.cc:359] Metadata directory not provided
I20260621 05:10:45.459028 10365 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 05:10:45.459216 10365 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 05:10:45.460412 10365 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782018616648578-3598-0/minicluster-data/master-2/data/instance:
uuid: "e6286f69e19545e883f56d392e7b8ea9"
format_stamp: "Formatted at 2026-06-21 05:10:45 on dist-test-slave-vrq1"
server_key: "e19903fad760d9372bbe20e226008456"
server_key_iv: "0365aaaa57e171e82aa778260566f896"
server_key_version: "encryptionkey@0"
I20260621 05:10:45.460822 10365 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782018616648578-3598-0/minicluster-data/master-2/wal/instance:
uuid: "e6286f69e19545e883f56d392e7b8ea9"
format_stamp: "Formatted at 2026-06-21 05:10:45 on dist-test-slave-vrq1"
server_key: "e19903fad760d9372bbe20e226008456"
server_key_iv: "0365aaaa57e171e82aa778260566f896"
server_key_version: "encryptionkey@0"
I20260621 05:10:45.462328 10365 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260621 05:10:45.463296 10380 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 05:10:45.463493 10365 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260621 05:10:45.463593 10365 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782018616648578-3598-0/minicluster-data/master-2/data,/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782018616648578-3598-0/minicluster-data/master-2/wal
uuid: "e6286f69e19545e883f56d392e7b8ea9"
format_stamp: "Formatted at 2026-06-21 05:10:45 on dist-test-slave-vrq1"
server_key: "e19903fad760d9372bbe20e226008456"
server_key_iv: "0365aaaa57e171e82aa778260566f896"
server_key_version: "encryptionkey@0"
I20260621 05:10:45.463701 10365 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782018616648578-3598-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782018616648578-3598-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782018616648578-3598-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 05:10:45.482231 10365 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 05:10:45.484740 10365 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 05:10:45.484895 10365 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 05:10:45.489490 10365 rpc_server.cc:307] RPC server started. Bound to: 127.3.131.188:32887
I20260621 05:10:45.489524 10432 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.131.188:32887 every 8 connection(s)
I20260621 05:10:45.489935 10365 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782018616648578-3598-0/minicluster-data/master-2/data/info.pb
I20260621 05:10:45.491101 10433 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 05:10:45.492821 10433 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.131.190" port: 33159 } has no permanent_uuid. Determining permanent_uuid...
I20260621 05:10:45.497378 3598 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWZOx2_/build/release/bin/kudu as pid 10365
Jun 21 05:10:45 dist-test-slave-vrq1 krb5kdc[9668](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782018645, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.131.188@KRBTEST.COM for oryx/127.3.131.190@KRBTEST.COM
I20260621 05:10:45.502715 9752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:10:45.493742 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53602 (local address 127.3.131.190:33159)
0621 05:10:45.493820 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0621 05:10:45.493824 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 05:10:45.493991 (+ 167us) server_negotiation.cc:407] Connection header received
0621 05:10:45.494451 (+ 460us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:10:45.494452 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:10:45.494464 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:10:45.494528 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:10:45.495264 (+ 736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:10:45.495797 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:10:45.496803 (+ 1006us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:10:45.496986 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:10:45.497043 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:10:45.499584 (+ 2541us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:10:45.499586 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:10:45.499586 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:10:45.499605 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:10:45.501022 (+ 1417us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:10:45.501641 (+ 619us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:10:45.501642 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:10:45.501643 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:10:45.501710 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:10:45.501962 (+ 252us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:10:45.501963 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:10:45.501964 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:10:45.502173 (+ 209us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:10:45.502255 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0621 05:10:45.502508 (+ 253us) server_negotiation.cc:299] Negotiation successful
0621 05:10:45.502588 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260621 05:10:45.502799 10435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:10:45.493641 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.131.190:33159 (local address 127.0.0.1:53602)
0621 05:10:45.493835 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0621 05:10:45.493879 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0621 05:10:45.494358 (+ 479us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:10:45.494610 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:10:45.494613 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:10:45.494901 (+ 288us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:10:45.495108 (+ 207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:10:45.495113 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:10:45.495885 (+ 772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:10:45.495885 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:10:45.496722 (+ 837us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:10:45.496724 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:10:45.497075 (+ 351us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:10:45.497076 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:10:45.497221 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:10:45.497774 (+ 553us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 05:10:45.497783 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:10:45.499485 (+ 1702us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 05:10:45.501162 (+ 1677us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:10:45.501166 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:10:45.501168 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:10:45.501451 (+ 283us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:10:45.501785 (+ 334us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:10:45.501786 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:10:45.501786 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:10:45.501890 (+ 104us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:10:45.502253 (+ 363us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 05:10:45.502259 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 05:10:45.502376 (+ 117us) client_negotiation.cc:769] Sending connection context
0621 05:10:45.502465 (+ 89us) client_negotiation.cc:240] Negotiation successful
0621 05:10:45.502601 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":150,"thread_start_us":115,"threads_started":1}
Jun 21 05:10:45 dist-test-slave-vrq1 krb5kdc[9668](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782018644, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.3.131.188@KRBTEST.COM
I20260621 05:10:45.503444 10433 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.131.189" port: 41383 } has no permanent_uuid. Determining permanent_uuid...
I20260621 05:10:45.507548 10436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:10:45.497969 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35402 (local address 127.3.131.188:32887)
0621 05:10:45.498149 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0621 05:10:45.498154 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 05:10:45.498172 (+ 18us) server_negotiation.cc:407] Connection header received
0621 05:10:45.498193 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:10:45.498195 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:10:45.498213 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:10:45.498299 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:10:45.498850 (+ 551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:10:45.499688 (+ 838us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:10:45.500712 (+ 1024us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:10:45.501003 (+ 291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:10:45.501059 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:10:45.504032 (+ 2973us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:10:45.504038 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:10:45.504038 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:10:45.504045 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:10:45.506048 (+ 2003us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:10:45.506439 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:10:45.506442 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:10:45.506442 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:10:45.506492 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:10:45.506774 (+ 282us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:10:45.506775 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:10:45.506775 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:10:45.506967 (+ 192us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:10:45.507248 (+ 281us) server_negotiation.cc:1035] Waiting for connection context
0621 05:10:45.507297 (+ 49us) server_negotiation.cc:299] Negotiation successful
0621 05:10:45.507372 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":149,"thread_start_us":97,"threads_started":1}
Jun 21 05:10:45 dist-test-slave-vrq1 krb5kdc[9668](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782018645, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.131.188@KRBTEST.COM for oryx/127.3.131.189@KRBTEST.COM
I20260621 05:10:45.511097 10362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:10:45.503830 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35216 (local address 127.3.131.189:41383)
0621 05:10:45.503873 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0621 05:10:45.503875 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:10:45.503882 (+ 7us) server_negotiation.cc:407] Connection header received
0621 05:10:45.503912 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:10:45.503912 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:10:45.503930 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:10:45.503975 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:10:45.504233 (+ 258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:10:45.505015 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:10:45.506139 (+ 1124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:10:45.506472 (+ 333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:10:45.506528 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:10:45.508470 (+ 1942us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:10:45.508471 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:10:45.508471 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:10:45.508478 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:10:45.509871 (+ 1393us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:10:45.510277 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:10:45.510278 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:10:45.510278 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:10:45.510368 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:10:45.510601 (+ 233us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:10:45.510601 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:10:45.510602 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:10:45.510735 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:10:45.510792 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0621 05:10:45.510961 (+ 169us) server_negotiation.cc:299] Negotiation successful
0621 05:10:45.511034 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260621 05:10:45.511116 10435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:10:45.503769 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.131.189:41383 (local address 127.0.0.1:35216)
0621 05:10:45.503814 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0621 05:10:45.503824 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0621 05:10:45.503858 (+ 34us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:10:45.504002 (+ 144us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:10:45.504003 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:10:45.504129 (+ 126us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:10:45.504171 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:10:45.504172 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:10:45.505134 (+ 962us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:10:45.505136 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:10:45.506002 (+ 866us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:10:45.506004 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:10:45.506544 (+ 540us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:10:45.506545 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:10:45.506671 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:10:45.506991 (+ 320us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 05:10:45.506996 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:10:45.508395 (+ 1399us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 05:10:45.509966 (+ 1571us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:10:45.509966 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:10:45.509967 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:10:45.510174 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:10:45.510464 (+ 290us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:10:45.510465 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:10:45.510465 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:10:45.510529 (+ 64us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:10:45.510817 (+ 288us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 05:10:45.510818 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 05:10:45.510862 (+ 44us) client_negotiation.cc:769] Sending connection context
0621 05:10:45.510930 (+ 68us) client_negotiation.cc:240] Negotiation successful
0621 05:10:45.511054 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32}
I20260621 05:10:45.511600 10433 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.131.188" port: 32887 } has no permanent_uuid. Determining permanent_uuid...
Jun 21 05:10:45 dist-test-slave-vrq1 krb5kdc[9668](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782018645, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.131.188@KRBTEST.COM for oryx/127.3.131.188@KRBTEST.COM
I20260621 05:10:45.519027 10435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:10:45.511875 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.131.188:32887 (local address 127.0.0.1:35412)
0621 05:10:45.511928 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0621 05:10:45.511939 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0621 05:10:45.512033 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:10:45.512125 (+ 92us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:10:45.512125 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:10:45.512231 (+ 106us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:10:45.512262 (+ 31us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:10:45.512263 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:10:45.512868 (+ 605us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:10:45.512869 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:10:45.513385 (+ 516us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:10:45.513386 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:10:45.513922 (+ 536us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:10:45.513923 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:10:45.513965 (+ 42us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:10:45.514387 (+ 422us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 05:10:45.514391 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:10:45.516225 (+ 1834us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 05:10:45.518190 (+ 1965us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:10:45.518191 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:10:45.518191 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:10:45.518366 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:10:45.518536 (+ 170us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:10:45.518537 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:10:45.518537 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:10:45.518579 (+ 42us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:10:45.518780 (+ 201us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 05:10:45.518781 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 05:10:45.518815 (+ 34us) client_negotiation.cc:769] Sending connection context
0621 05:10:45.518867 (+ 52us) client_negotiation.cc:240] Negotiation successful
0621 05:10:45.518945 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":36}
I20260621 05:10:45.519080 10436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:10:45.511933 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35412 (local address 127.3.131.188:32887)
0621 05:10:45.511976 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0621 05:10:45.511978 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:10:45.511984 (+ 6us) server_negotiation.cc:407] Connection header received
0621 05:10:45.512073 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:10:45.512074 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:10:45.512082 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:10:45.512123 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:10:45.512312 (+ 189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:10:45.512790 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:10:45.513479 (+ 689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:10:45.513821 (+ 342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:10:45.513925 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:10:45.516310 (+ 2385us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:10:45.516312 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:10:45.516313 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:10:45.516320 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:10:45.518104 (+ 1784us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:10:45.518430 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:10:45.518431 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:10:45.518431 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:10:45.518492 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:10:45.518627 (+ 135us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:10:45.518628 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:10:45.518628 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:10:45.518726 (+ 98us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:10:45.518771 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0621 05:10:45.518900 (+ 129us) server_negotiation.cc:299] Negotiation successful
0621 05:10:45.518990 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":26,"server-negotiator.queue_time_us":19}
I20260621 05:10:45.520534 10433 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e6286f69e19545e883f56d392e7b8ea9: Bootstrap starting.
I20260621 05:10:45.521378 10433 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e6286f69e19545e883f56d392e7b8ea9: Neither blocks nor log segments found. Creating new log.
I20260621 05:10:45.521677 10433 log.cc:824] T 00000000000000000000000000000000 P e6286f69e19545e883f56d392e7b8ea9: Log is configured to *not* fsync() on all Append() calls
I20260621 05:10:45.522462 10433 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e6286f69e19545e883f56d392e7b8ea9: No bootstrap required, opened a new log
I20260621 05:10:45.524034 10433 raft_consensus.cc:348] T 00000000000000000000000000000000 P e6286f69e19545e883f56d392e7b8ea9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08adad2fec7f4618bfbb0504365c5ecc" member_type: VOTER last_known_addr { host: "127.3.131.190" port: 33159 } } peers { permanent_uuid: "65fa57b1227645fa8b95922a4da9e2ef" member_type: VOTER last_known_addr { host: "127.3.131.189" port: 41383 } } peers { permanent_uuid: "e6286f69e19545e883f56d392e7b8ea9" member_type: VOTER last_known_addr { host: "127.3.131.188" port: 32887 } }
I20260621 05:10:45.524186 10433 raft_consensus.cc:374] T 00000000000000000000000000000000 P e6286f69e19545e883f56d392e7b8ea9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 05:10:45.524235 10433 raft_consensus.cc:729] T 00000000000000000000000000000000 P e6286f69e19545e883f56d392e7b8ea9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e6286f69e19545e883f56d392e7b8ea9, State: Initialized, Role: FOLLOWER
I20260621 05:10:45.524374 10433 consensus_queue.cc:260] T 00000000000000000000000000000000 P e6286f69e19545e883f56d392e7b8ea9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08adad2fec7f4618bfbb0504365c5ecc" member_type: VOTER last_known_addr { host: "127.3.131.190" port: 33159 } } peers { permanent_uuid: "65fa57b1227645fa8b95922a4da9e2ef" member_type: VOTER last_known_addr { host: "127.3.131.189" port: 41383 } } peers { permanent_uuid: "e6286f69e19545e883f56d392e7b8ea9" member_type: VOTER last_known_addr { host: "127.3.131.188" port: 32887 } }
I20260621 05:10:45.524686 10433 sys_catalog.cc:564] T 00000000000000000000000000000000 P e6286f69e19545e883f56d392e7b8ea9 [sys.catalog]: configured and running, proceeding with master startup.
I20260621 05:10:45.524682 10438 sys_catalog.cc:455] T 00000000000000000000000000000000 P e6286f69e19545e883f56d392e7b8ea9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08adad2fec7f4618bfbb0504365c5ecc" member_type: VOTER last_known_addr { host: "127.3.131.190" port: 33159 } } peers { permanent_uuid: "65fa57b1227645fa8b95922a4da9e2ef" member_type: VOTER last_known_addr { host: "127.3.131.189" port: 41383 } } peers { permanent_uuid: "e6286f69e19545e883f56d392e7b8ea9" member_type: VOTER last_known_addr { host: "127.3.131.188" port: 32887 } } }
I20260621 05:10:45.524838 10438 sys_catalog.cc:458] T 00000000000000000000000000000000 P e6286f69e19545e883f56d392e7b8ea9 [sys.catalog]: This master's current role is: FOLLOWER
W20260621 05:10:45.526476 10449 catalog_manager.cc:1558] T 00000000000000000000000000000000 P e6286f69e19545e883f56d392e7b8ea9: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260621 05:10:45.526548 10449 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260621 05:10:45.528658 10438 raft_consensus.cc:482] T 00000000000000000000000000000000 P e6286f69e19545e883f56d392e7b8ea9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260621 05:10:45.528764 10438 raft_consensus.cc:504] T 00000000000000000000000000000000 P e6286f69e19545e883f56d392e7b8ea9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08adad2fec7f4618bfbb0504365c5ecc" member_type: VOTER last_known_addr { host: "127.3.131.190" port: 33159 } } peers { permanent_uuid: "65fa57b1227645fa8b95922a4da9e2ef" member_type: VOTER last_known_addr { host: "127.3.131.189" port: 41383 } } peers { permanent_uuid: "e6286f69e19545e883f56d392e7b8ea9" member_type: VOTER last_known_addr { host: "127.3.131.188" port: 32887 } }
I20260621 05:10:45.529018 10438 leader_election.cc:288] T 00000000000000000000000000000000 P e6286f69e19545e883f56d392e7b8ea9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 08adad2fec7f4618bfbb0504365c5ecc (127.3.131.190:33159), 65fa57b1227645fa8b95922a4da9e2ef (127.3.131.189:41383)
I20260621 05:10:45.529223 10321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e6286f69e19545e883f56d392e7b8ea9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "65fa57b1227645fa8b95922a4da9e2ef" is_pre_election: true
I20260621 05:10:45.529332 10321 raft_consensus.cc:1799] T 00000000000000000000000000000000 P 65fa57b1227645fa8b95922a4da9e2ef [term 1 LEARNER]: Handling vote request from an unknown peer e6286f69e19545e883f56d392e7b8ea9
I20260621 05:10:45.529299 9723 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e6286f69e19545e883f56d392e7b8ea9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "08adad2fec7f4618bfbb0504365c5ecc" is_pre_election: true
I20260621 05:10:45.529376 10321 raft_consensus.cc:2396] T 00000000000000000000000000000000 P 65fa57b1227645fa8b95922a4da9e2ef [term 1 LEARNER]: Leader pre-election vote request: Denying vote to candidate e6286f69e19545e883f56d392e7b8ea9 for term 1 because replica has last-logged OpId of term: 1 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 0 index: 0.
I20260621 05:10:45.529477 10365 master_runner.cc:428] Detected that this master e6286f69e19545e883f56d392e7b8ea9 is joining an existing cluster
I20260621 05:10:45.529523 10365 master_runner.cc:432] Initiating AddMaster RPC to add 127.3.131.188:32887
I20260621 05:10:45.529867 10382 leader_election.cc:302] T 00000000000000000000000000000000 P e6286f69e19545e883f56d392e7b8ea9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e6286f69e19545e883f56d392e7b8ea9; no voters: 08adad2fec7f4618bfbb0504365c5ecc, 65fa57b1227645fa8b95922a4da9e2ef
I20260621 05:10:45.530124 10438 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e6286f69e19545e883f56d392e7b8ea9 [term 0 FOLLOWER]: Advancing to term 1
I20260621 05:10:45.531268 10438 raft_consensus.cc:2726] T 00000000000000000000000000000000 P e6286f69e19545e883f56d392e7b8ea9 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260621 05:10:45.538985 10362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:10:45.531152 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35226 (local address 127.3.131.189:41383)
0621 05:10:45.531234 (+ 82us) server_negotiation.cc:206] Beginning negotiation
0621 05:10:45.531235 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0621 05:10:45.531393 (+ 158us) server_negotiation.cc:407] Connection header received
0621 05:10:45.531697 (+ 304us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:10:45.531698 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:10:45.531708 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:10:45.531736 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:10:45.532516 (+ 780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:10:45.533281 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:10:45.534277 (+ 996us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:10:45.534545 (+ 268us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:10:45.534639 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:10:45.536210 (+ 1571us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:10:45.536212 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:10:45.536212 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:10:45.536219 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:10:45.537732 (+ 1513us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:10:45.538183 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:10:45.538184 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:10:45.538184 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:10:45.538266 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:10:45.538495 (+ 229us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:10:45.538495 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:10:45.538495 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:10:45.538618 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:10:45.538673 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0621 05:10:45.538833 (+ 160us) server_negotiation.cc:299] Negotiation successful
0621 05:10:45.538904 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260621 05:10:45.539045 10456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:10:45.530797 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.131.189:41383 (local address 127.0.0.1:35226)
0621 05:10:45.531334 (+ 537us) negotiation.cc:107] Waiting for socket to connect
0621 05:10:45.531344 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0621 05:10:45.531540 (+ 196us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:10:45.532192 (+ 652us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:10:45.532192 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:10:45.532375 (+ 183us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:10:45.532427 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:10:45.532429 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:10:45.533379 (+ 950us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:10:45.533379 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:10:45.534204 (+ 825us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:10:45.534207 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:10:45.534636 (+ 429us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:10:45.534637 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:10:45.534706 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:10:45.535586 (+ 880us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 05:10:45.535590 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:10:45.536076 (+ 486us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 05:10:45.537821 (+ 1745us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:10:45.537822 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:10:45.537823 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:10:45.538083 (+ 260us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:10:45.538343 (+ 260us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:10:45.538344 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:10:45.538344 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:10:45.538429 (+ 85us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:10:45.538684 (+ 255us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 05:10:45.538685 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 05:10:45.538735 (+ 50us) client_negotiation.cc:769] Sending connection context
0621 05:10:45.538817 (+ 82us) client_negotiation.cc:240] Negotiation successful
0621 05:10:45.538927 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":521,"thread_start_us":49,"threads_started":1}
I20260621 05:10:45.559271 3598 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWZOx2_/build/release/bin/kudu with pid 9775
I20260621 05:10:45.563776 10436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:10:45.531008 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35416 (local address 127.3.131.188:32887)
0621 05:10:45.531048 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0621 05:10:45.531050 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:10:45.531476 (+ 426us) server_negotiation.cc:407] Connection header received
0621 05:10:45.531606 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:10:45.531607 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:10:45.531615 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:10:45.531641 (+ 26us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:10:45.532168 (+ 527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:10:45.532827 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:10:45.533756 (+ 929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:10:45.534021 (+ 265us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:10:45.534058 (+ 37us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:10:45.561056 (+ 26998us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:10:45.561058 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:10:45.561058 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:10:45.561064 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:10:45.562625 (+ 1561us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:10:45.562969 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:10:45.562969 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:10:45.562970 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:10:45.563013 (+ 43us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:10:45.563225 (+ 212us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:10:45.563226 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:10:45.563226 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:10:45.563361 (+ 135us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:10:45.563402 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0621 05:10:45.563570 (+ 168us) server_negotiation.cc:299] Negotiation successful
0621 05:10:45.563677 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260621 05:10:45.564391 10457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:10:45.530961 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.131.188:32887 (local address 127.0.0.1:35416)
0621 05:10:45.531431 (+ 470us) negotiation.cc:107] Waiting for socket to connect
0621 05:10:45.531439 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0621 05:10:45.531498 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:10:45.531818 (+ 320us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:10:45.531819 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:10:45.532007 (+ 188us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:10:45.532074 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:10:45.532076 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:10:45.532911 (+ 835us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:10:45.532912 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:10:45.533685 (+ 773us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:10:45.533686 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:10:45.534122 (+ 436us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:10:45.534123 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:10:45.534185 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:10:45.560554 (+ 26369us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 05:10:45.560560 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:10:45.560962 (+ 402us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 05:10:45.562713 (+ 1751us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:10:45.562714 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:10:45.562714 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:10:45.562902 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:10:45.563074 (+ 172us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:10:45.563074 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:10:45.563075 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:10:45.563137 (+ 62us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:10:45.563440 (+ 303us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 05:10:45.563441 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 05:10:45.563476 (+ 35us) client_negotiation.cc:769] Sending connection context
0621 05:10:45.564177 (+ 701us) client_negotiation.cc:240] Negotiation successful
0621 05:10:45.564286 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":459,"thread_start_us":70,"threads_started":1}
I20260621 05:10:45.564941 9752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:10:45.531270 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53612 (local address 127.3.131.190:33159)
0621 05:10:45.531338 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0621 05:10:45.531341 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0621 05:10:45.531350 (+ 9us) server_negotiation.cc:407] Connection header received
0621 05:10:45.531374 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:10:45.531375 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:10:45.531387 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:10:45.531445 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:10:45.531803 (+ 358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:10:45.532437 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:10:45.533492 (+ 1055us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:10:45.533812 (+ 320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:10:45.533849 (+ 37us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:10:45.560815 (+ 26966us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:10:45.560818 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:10:45.560818 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:10:45.560829 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:10:45.563419 (+ 2590us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:10:45.563899 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:10:45.563900 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:10:45.563901 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:10:45.563982 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:10:45.564319 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:10:45.564320 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:10:45.564321 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:10:45.564472 (+ 151us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:10:45.564515 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0621 05:10:45.564712 (+ 197us) server_negotiation.cc:299] Negotiation successful
0621 05:10:45.564803 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260621 05:10:45.565366 10455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:10:45.530603 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.131.190:33159 (local address 127.0.0.1:53612)
0621 05:10:45.530754 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0621 05:10:45.530767 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0621 05:10:45.530823 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:10:45.531465 (+ 642us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:10:45.531466 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:10:45.531649 (+ 183us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:10:45.531715 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:10:45.531717 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:10:45.532534 (+ 817us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:10:45.532535 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:10:45.533401 (+ 866us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:10:45.533403 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:10:45.533922 (+ 519us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:10:45.533923 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:10:45.533986 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:10:45.534629 (+ 643us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 05:10:45.534635 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:10:45.560403 (+ 25768us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 05:10:45.563524 (+ 3121us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:10:45.563525 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:10:45.563525 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:10:45.563807 (+ 282us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:10:45.564178 (+ 371us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:10:45.564179 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:10:45.564180 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:10:45.564248 (+ 68us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:10:45.564558 (+ 310us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 05:10:45.564559 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 05:10:45.564605 (+ 46us) client_negotiation.cc:769] Sending connection context
0621 05:10:45.565062 (+ 457us) client_negotiation.cc:240] Negotiation successful
0621 05:10:45.565200 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":73,"threads_started":1}
I20260621 05:10:45.566356 3598 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWZOx2_/build/release/bin/kudu
/tmp/dist-test-taskWZOx2_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782018616648578-3598-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782018616648578-3598-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782018616648578-3598-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782018616648578-3598-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.131.129:38671
--local_ip_for_outbound_sockets=127.3.131.129
--tserver_master_addrs=127.3.131.190:33159
--webserver_port=37023
--webserver_interface=127.3.131.129
--keytab_file=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782018616648578-3598-0/krb5kdc/oryx_127.3.131.129.keytab
--principal=oryx/127.3.131.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.131.148:35089
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.3.131.190:33159,127.3.131.189:41383
--tserver_master_addrs=127.3.131.190:33159,127.3.131.189:41383,127.3.131.188:32887 with env {KRB5CCNAME=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782018616648578-3598-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782018616648578-3598-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782018616648578-3598-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 21 05:10:45 dist-test-slave-vrq1 krb5kdc[9668](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782018644, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.131.190@KRBTEST.COM for oryx/127.3.131.188@KRBTEST.COM
I20260621 05:10:45.576254 10436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:10:45.567957 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35420 (local address 127.3.131.188:32887)
0621 05:10:45.568025 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0621 05:10:45.568027 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:10:45.568036 (+ 9us) server_negotiation.cc:407] Connection header received
0621 05:10:45.568070 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:10:45.568071 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:10:45.568080 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:10:45.568150 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:10:45.568518 (+ 368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:10:45.569249 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:10:45.570358 (+ 1109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:10:45.570712 (+ 354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:10:45.570786 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:10:45.573167 (+ 2381us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:10:45.573169 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:10:45.573169 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:10:45.573177 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:10:45.574885 (+ 1708us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:10:45.575348 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:10:45.575349 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:10:45.575350 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:10:45.575414 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:10:45.575645 (+ 231us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:10:45.575646 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:10:45.575646 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:10:45.575797 (+ 151us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:10:45.575867 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0621 05:10:45.576068 (+ 201us) server_negotiation.cc:299] Negotiation successful
0621 05:10:45.576158 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260621 05:10:45.576295 10271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:10:45.567711 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.131.188:32887 (local address 127.0.0.1:35420)
0621 05:10:45.567818 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0621 05:10:45.567831 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0621 05:10:45.567874 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:10:45.568175 (+ 301us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:10:45.568177 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:10:45.568368 (+ 191us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:10:45.568434 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:10:45.568436 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:10:45.569337 (+ 901us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:10:45.569339 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:10:45.570250 (+ 911us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:10:45.570254 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:10:45.570797 (+ 543us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:10:45.570798 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:10:45.570878 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:10:45.571387 (+ 509us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 05:10:45.571392 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:10:45.573050 (+ 1658us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 05:10:45.574983 (+ 1933us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:10:45.574985 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:10:45.574986 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:10:45.575266 (+ 280us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:10:45.575494 (+ 228us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:10:45.575495 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:10:45.575495 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:10:45.575570 (+ 75us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:10:45.575875 (+ 305us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 05:10:45.575876 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 05:10:45.575937 (+ 61us) client_negotiation.cc:769] Sending connection context
0621 05:10:45.576023 (+ 86us) client_negotiation.cc:240] Negotiation successful
0621 05:10:45.576149 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":84}
I20260621 05:10:45.577286 9713 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.3.131.188:32887: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "e6286f69e19545e883f56d392e7b8ea9"
member_type: NON_VOTER
last_known_addr {
host: "127.3.131.188"
port: 32887
}
attrs {
promote: true
}
}
dest_uuid: "08adad2fec7f4618bfbb0504365c5ecc"
cas_config_opid_index: 5
I20260621 05:10:45.577586 9713 consensus_queue.cc:237] T 00000000000000000000000000000000 P 08adad2fec7f4618bfbb0504365c5ecc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "08adad2fec7f4618bfbb0504365c5ecc" member_type: VOTER last_known_addr { host: "127.3.131.190" port: 33159 } } peers { permanent_uuid: "65fa57b1227645fa8b95922a4da9e2ef" member_type: NON_VOTER last_known_addr { host: "127.3.131.189" port: 41383 } attrs { promote: true } } peers { permanent_uuid: "e6286f69e19545e883f56d392e7b8ea9" member_type: NON_VOTER last_known_addr { host: "127.3.131.188" port: 32887 } attrs { promote: true } }
I20260621 05:10:45.577978 10272 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 08adad2fec7f4618bfbb0504365c5ecc [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER e6286f69e19545e883f56d392e7b8ea9 (127.3.131.188) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "08adad2fec7f4618bfbb0504365c5ecc" member_type: VOTER last_known_addr { host: "127.3.131.190" port: 33159 } } peers { permanent_uuid: "65fa57b1227645fa8b95922a4da9e2ef" member_type: NON_VOTER last_known_addr { host: "127.3.131.189" port: 41383 } attrs { promote: true } } peers { permanent_uuid: "e6286f69e19545e883f56d392e7b8ea9" member_type: NON_VOTER last_known_addr { host: "127.3.131.188" port: 32887 } attrs { promote: true } } }
I20260621 05:10:45.578953 10272 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.3.131.188:32887
I20260621 05:10:45.579030 10272 sys_catalog.cc:455] T 00000000000000000000000000000000 P 08adad2fec7f4618bfbb0504365c5ecc [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "08adad2fec7f4618bfbb0504365c5ecc" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "08adad2fec7f4618bfbb0504365c5ecc" member_type: VOTER last_known_addr { host: "127.3.131.190" port: 33159 } } peers { permanent_uuid: "65fa57b1227645fa8b95922a4da9e2ef" member_type: NON_VOTER last_known_addr { host: "127.3.131.189" port: 41383 } attrs { promote: true } } peers { permanent_uuid: "e6286f69e19545e883f56d392e7b8ea9" member_type: NON_VOTER last_known_addr { host: "127.3.131.188" port: 32887 } attrs { promote: true } } }
I20260621 05:10:45.579095 10272 sys_catalog.cc:458] T 00000000000000000000000000000000 P 08adad2fec7f4618bfbb0504365c5ecc [sys.catalog]: This master's current role is: LEADER
I20260621 05:10:45.579434 10365 master.cc:537] Master@127.3.131.188:32887 shutting down...
W20260621 05:10:45.580556 9699 consensus_peers.cc:487] T 00000000000000000000000000000000 P 08adad2fec7f4618bfbb0504365c5ecc -> Peer e6286f69e19545e883f56d392e7b8ea9 (127.3.131.188:32887): Couldn't send request to peer e6286f69e19545e883f56d392e7b8ea9. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260621 05:10:45.582640 10365 raft_consensus.cc:2229] T 00000000000000000000000000000000 P e6286f69e19545e883f56d392e7b8ea9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260621 05:10:45.582726 10365 raft_consensus.cc:2258] T 00000000000000000000000000000000 P e6286f69e19545e883f56d392e7b8ea9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 05:10:45.582764 10365 tablet_replica.cc:330] T 00000000000000000000000000000000 P e6286f69e19545e883f56d392e7b8ea9: stopping tablet replica
I20260621 05:10:45.584527 10362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:10:45.579630 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35236 (local address 127.3.131.189:41383)
0621 05:10:45.579720 (+ 90us) server_negotiation.cc:206] Beginning negotiation
0621 05:10:45.579722 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:10:45.580434 (+ 712us) server_negotiation.cc:407] Connection header received
0621 05:10:45.580725 (+ 291us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:10:45.580727 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:10:45.580750 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:10:45.580789 (+ 39us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 05:10:45.580985 (+ 196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:10:45.581749 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:10:45.583243 (+ 1494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:10:45.583854 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:10:45.583902 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:10:45.583975 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0621 05:10:45.584392 (+ 417us) server_negotiation.cc:299] Negotiation successful
0621 05:10:45.584443 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260621 05:10:45.584731 10271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:10:45.579378 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.131.189:41383 (local address 127.0.0.1:35236)
0621 05:10:45.580382 (+ 1004us) negotiation.cc:107] Waiting for socket to connect
0621 05:10:45.580393 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0621 05:10:45.580640 (+ 247us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:10:45.580870 (+ 230us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:10:45.580871 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:10:45.580872 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 05:10:45.580917 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:10:45.580919 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:10:45.581853 (+ 934us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:10:45.581854 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:10:45.583119 (+ 1265us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:10:45.583122 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:10:45.584207 (+ 1085us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:10:45.584208 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:10:45.584281 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:10:45.584307 (+ 26us) client_negotiation.cc:769] Sending connection context
0621 05:10:45.584606 (+ 299us) client_negotiation.cc:240] Negotiation successful
0621 05:10:45.584664 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":988}
I20260621 05:10:45.585076 10321 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 65fa57b1227645fa8b95922a4da9e2ef [term 1 LEARNER]: Refusing update from remote peer 08adad2fec7f4618bfbb0504365c5ecc: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260621 05:10:45.585368 10273 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 08adad2fec7f4618bfbb0504365c5ecc [LEADER]: Connected to new peer: Peer: permanent_uuid: "65fa57b1227645fa8b95922a4da9e2ef" member_type: NON_VOTER last_known_addr { host: "127.3.131.189" port: 41383 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260621 05:10:45.585970 10321 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 65fa57b1227645fa8b95922a4da9e2ef [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER e6286f69e19545e883f56d392e7b8ea9 (127.3.131.188) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "08adad2fec7f4618bfbb0504365c5ecc" member_type: VOTER last_known_addr { host: "127.3.131.190" port: 33159 } } peers { permanent_uuid: "65fa57b1227645fa8b95922a4da9e2ef" member_type: NON_VOTER last_known_addr { host: "127.3.131.189" port: 41383 } attrs { promote: true } } peers { permanent_uuid: "e6286f69e19545e883f56d392e7b8ea9" member_type: NON_VOTER last_known_addr { host: "127.3.131.188" port: 32887 } attrs { promote: true } } }
I20260621 05:10:45.587113 10350 sys_catalog.cc:455] T 00000000000000000000000000000000 P 65fa57b1227645fa8b95922a4da9e2ef [sys.catalog]: SysCatalogTable state changed. Reason: New leader 08adad2fec7f4618bfbb0504365c5ecc. Latest consensus state: current_term: 1 leader_uuid: "08adad2fec7f4618bfbb0504365c5ecc" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "08adad2fec7f4618bfbb0504365c5ecc" member_type: VOTER last_known_addr { host: "127.3.131.190" port: 33159 } } peers { permanent_uuid: "65fa57b1227645fa8b95922a4da9e2ef" member_type: NON_VOTER last_known_addr { host: "127.3.131.189" port: 41383 } attrs { promote: true } } peers { permanent_uuid: "e6286f69e19545e883f56d392e7b8ea9" member_type: NON_VOTER last_known_addr { host: "127.3.131.188" port: 32887 } attrs { promote: true } } }
I20260621 05:10:45.587421 10350 sys_catalog.cc:458] T 00000000000000000000000000000000 P 65fa57b1227645fa8b95922a4da9e2ef [sys.catalog]: This master's current role is: LEARNER
I20260621 05:10:45.587463 10350 sys_catalog.cc:455] T 00000000000000000000000000000000 P 65fa57b1227645fa8b95922a4da9e2ef [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "08adad2fec7f4618bfbb0504365c5ecc" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "08adad2fec7f4618bfbb0504365c5ecc" member_type: VOTER last_known_addr { host: "127.3.131.190" port: 33159 } } peers { permanent_uuid: "65fa57b1227645fa8b95922a4da9e2ef" member_type: NON_VOTER last_known_addr { host: "127.3.131.189" port: 41383 } attrs { promote: true } } peers { permanent_uuid: "e6286f69e19545e883f56d392e7b8ea9" member_type: NON_VOTER last_known_addr { host: "127.3.131.188" port: 32887 } attrs { promote: true } } }
I20260621 05:10:45.587515 10350 sys_catalog.cc:458] T 00000000000000000000000000000000 P 65fa57b1227645fa8b95922a4da9e2ef [sys.catalog]: This master's current role is: LEARNER
I20260621 05:10:45.587764 10273 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 08adad2fec7f4618bfbb0504365c5ecc: attempting to promote NON_VOTER 65fa57b1227645fa8b95922a4da9e2ef to VOTER
I20260621 05:10:45.587954 10273 consensus_queue.cc:237] T 00000000000000000000000000000000 P 08adad2fec7f4618bfbb0504365c5ecc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "08adad2fec7f4618bfbb0504365c5ecc" member_type: VOTER last_known_addr { host: "127.3.131.190" port: 33159 } } peers { permanent_uuid: "65fa57b1227645fa8b95922a4da9e2ef" member_type: VOTER last_known_addr { host: "127.3.131.189" port: 41383 } attrs { promote: false } } peers { permanent_uuid: "e6286f69e19545e883f56d392e7b8ea9" member_type: NON_VOTER last_known_addr { host: "127.3.131.188" port: 32887 } attrs { promote: true } }
I20260621 05:10:45.588447 10271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:10:45.588332 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.131.188:32887 (local address 127.0.0.1:35430)
0621 05:10:45.588376 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0621 05:10:45.588417 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.131.188:32887: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
I20260621 05:10:45.588814 10271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:10:45.588724 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.131.188:32887 (local address 127.0.0.1:35444)
0621 05:10:45.588766 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0621 05:10:45.588787 (+ 21us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.131.188:32887: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
I20260621 05:10:45.588908 10321 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 65fa57b1227645fa8b95922a4da9e2ef [term 1 LEARNER]: Refusing update from remote peer 08adad2fec7f4618bfbb0504365c5ecc: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
W20260621 05:10:45.588941 9699 consensus_peers.cc:487] T 00000000000000000000000000000000 P 08adad2fec7f4618bfbb0504365c5ecc -> Peer e6286f69e19545e883f56d392e7b8ea9 (127.3.131.188:32887): Couldn't send request to peer e6286f69e19545e883f56d392e7b8ea9. Status: Network error: Client connection negotiation failed: client connection to 127.3.131.188:32887: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260621 05:10:45.589079 10272 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 08adad2fec7f4618bfbb0504365c5ecc [LEADER]: Connected to new peer: Peer: permanent_uuid: "65fa57b1227645fa8b95922a4da9e2ef" member_type: VOTER last_known_addr { host: "127.3.131.189" port: 41383 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260621 05:10:45.589757 10461 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 08adad2fec7f4618bfbb0504365c5ecc [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 65fa57b1227645fa8b95922a4da9e2ef (127.3.131.189) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "08adad2fec7f4618bfbb0504365c5ecc" member_type: VOTER last_known_addr { host: "127.3.131.190" port: 33159 } } peers { permanent_uuid: "65fa57b1227645fa8b95922a4da9e2ef" member_type: VOTER last_known_addr { host: "127.3.131.189" port: 41383 } attrs { promote: false } } peers { permanent_uuid: "e6286f69e19545e883f56d392e7b8ea9" member_type: NON_VOTER last_known_addr { host: "127.3.131.188" port: 32887 } attrs { promote: true } } }
I20260621 05:10:45.589821 10321 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 65fa57b1227645fa8b95922a4da9e2ef [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 65fa57b1227645fa8b95922a4da9e2ef (127.3.131.189) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "08adad2fec7f4618bfbb0504365c5ecc" member_type: VOTER last_known_addr { host: "127.3.131.190" port: 33159 } } peers { permanent_uuid: "65fa57b1227645fa8b95922a4da9e2ef" member_type: VOTER last_known_addr { host: "127.3.131.189" port: 41383 } attrs { promote: false } } peers { permanent_uuid: "e6286f69e19545e883f56d392e7b8ea9" member_type: NON_VOTER last_known_addr { host: "127.3.131.188" port: 32887 } attrs { promote: true } } }
I20260621 05:10:45.590982 10460 sys_catalog.cc:455] T 00000000000000000000000000000000 P 65fa57b1227645fa8b95922a4da9e2ef [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "08adad2fec7f4618bfbb0504365c5ecc" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "08adad2fec7f4618bfbb0504365c5ecc" member_type: VOTER last_known_addr { host: "127.3.131.190" port: 33159 } } peers { permanent_uuid: "65fa57b1227645fa8b95922a4da9e2ef" member_type: VOTER last_known_addr { host: "127.3.131.189" port: 41383 } attrs { promote: false } } peers { permanent_uuid: "e6286f69e19545e883f56d392e7b8ea9" member_type: NON_VOTER last_known_addr { host: "127.3.131.188" port: 32887 } attrs { promote: true } } }
I20260621 05:10:45.591112 10460 sys_catalog.cc:458] T 00000000000000000000000000000000 P 65fa57b1227645fa8b95922a4da9e2ef [sys.catalog]: This master's current role is: FOLLOWER
I20260621 05:10:45.591316 10272 sys_catalog.cc:455] T 00000000000000000000000000000000 P 08adad2fec7f4618bfbb0504365c5ecc [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "08adad2fec7f4618bfbb0504365c5ecc" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "08adad2fec7f4618bfbb0504365c5ecc" member_type: VOTER last_known_addr { host: "127.3.131.190" port: 33159 } } peers { permanent_uuid: "65fa57b1227645fa8b95922a4da9e2ef" member_type: VOTER last_known_addr { host: "127.3.131.189" port: 41383 } attrs { promote: false } } peers { permanent_uuid: "e6286f69e19545e883f56d392e7b8ea9" member_type: NON_VOTER last_known_addr { host: "127.3.131.188" port: 32887 } attrs { promote: true } } }
I20260621 05:10:45.591414 10272 sys_catalog.cc:458] T 00000000000000000000000000000000 P 08adad2fec7f4618bfbb0504365c5ecc [sys.catalog]: This master's current role is: LEADER
I20260621 05:10:45.591496 10461 sys_catalog.cc:455] T 00000000000000000000000000000000 P 08adad2fec7f4618bfbb0504365c5ecc [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "08adad2fec7f4618bfbb0504365c5ecc" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "08adad2fec7f4618bfbb0504365c5ecc" member_type: VOTER last_known_addr { host: "127.3.131.190" port: 33159 } } peers { permanent_uuid: "65fa57b1227645fa8b95922a4da9e2ef" member_type: VOTER last_known_addr { host: "127.3.131.189" port: 41383 } attrs { promote: false } } peers { permanent_uuid: "e6286f69e19545e883f56d392e7b8ea9" member_type: NON_VOTER last_known_addr { host: "127.3.131.188" port: 32887 } attrs { promote: true } } }
I20260621 05:10:45.591550 10461 sys_catalog.cc:458] T 00000000000000000000000000000000 P 08adad2fec7f4618bfbb0504365c5ecc [sys.catalog]: This master's current role is: LEADER
I20260621 05:10:45.596858 10365 master.cc:552] Master@127.3.131.188:32887 shutdown complete.
I20260621 05:10:45.596917 10365 master_runner.cc:305] Clearing existing system tablet
I20260621 05:10:45.597046 10365 fs_manager.cc:359] Metadata directory not provided
I20260621 05:10:45.597102 10365 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 05:10:45.597882 10365 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.002s
I20260621 05:10:45.598454 10464 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 05:10:45.598562 10365 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782018616648578-3598-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 05:10:45.598608 10365 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260621 05:10:45.598629 10365 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782018616648578-3598-0/minicluster-data/master-2/data,/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782018616648578-3598-0/minicluster-data/master-2/wal
uuid: "e6286f69e19545e883f56d392e7b8ea9"
format_stamp: "Formatted at 2026-06-21 05:10:45 on dist-test-slave-vrq1"
server_key: "e19903fad760d9372bbe20e226008456"
server_key_iv: "0365aaaa57e171e82aa778260566f896"
server_key_version: "encryptionkey@0"
I20260621 05:10:45.599422 10365 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P e6286f69e19545e883f56d392e7b8ea9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 05:10:45.601671 10365 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P e6286f69e19545e883f56d392e7b8ea9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260621 05:10:45.601753 10365 log.cc:1196] T 00000000000000000000000000000000 P e6286f69e19545e883f56d392e7b8ea9: Deleting WAL directory at /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782018616648578-3598-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260621 05:10:45.601953 10365 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P e6286f69e19545e883f56d392e7b8ea9: Deleting consensus metadata
I20260621 05:10:45.602268 10365 master_runner.cc:315] Copying system tablet from 127.3.131.190:33159
I20260621 05:10:45.602939 10365 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P e6286f69e19545e883f56d392e7b8ea9: tablet copy: Beginning tablet copy session from remote peer at address 127.3.131.190:33159
I20260621 05:10:45.620048 10470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:10:45.612535 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.131.190:33159 (local address 127.0.0.1:53628)
0621 05:10:45.612707 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0621 05:10:45.612719 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0621 05:10:45.612786 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:10:45.612957 (+ 171us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:10:45.612958 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:10:45.613187 (+ 229us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:10:45.613248 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:10:45.613250 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:10:45.614234 (+ 984us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:10:45.614235 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:10:45.615304 (+ 1069us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:10:45.615307 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:10:45.615834 (+ 527us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:10:45.615835 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:10:45.615897 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:10:45.616465 (+ 568us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 05:10:45.616471 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:10:45.616890 (+ 419us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 05:10:45.618805 (+ 1915us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:10:45.618806 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:10:45.618807 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:10:45.619034 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:10:45.619309 (+ 275us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:10:45.619309 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:10:45.619310 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:10:45.619379 (+ 69us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:10:45.619700 (+ 321us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 05:10:45.619701 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 05:10:45.619755 (+ 54us) client_negotiation.cc:769] Sending connection context
0621 05:10:45.619827 (+ 72us) client_negotiation.cc:240] Negotiation successful
0621 05:10:45.619939 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":150,"thread_start_us":91,"threads_started":1}
I20260621 05:10:45.620054 9752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:10:45.612614 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53628 (local address 127.3.131.190:33159)
0621 05:10:45.612671 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0621 05:10:45.612673 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:10:45.612792 (+ 119us) server_negotiation.cc:407] Connection header received
0621 05:10:45.612838 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:10:45.612839 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:10:45.612851 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:10:45.612938 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:10:45.613349 (+ 411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:10:45.614134 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:10:45.615418 (+ 1284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:10:45.615739 (+ 321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:10:45.615821 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:10:45.616994 (+ 1173us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:10:45.616996 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:10:45.616996 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:10:45.617005 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:10:45.618712 (+ 1707us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:10:45.619123 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:10:45.619125 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:10:45.619125 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:10:45.619234 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:10:45.619458 (+ 224us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:10:45.619459 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:10:45.619459 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:10:45.619622 (+ 163us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:10:45.619700 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0621 05:10:45.619876 (+ 176us) server_negotiation.cc:299] Negotiation successful
0621 05:10:45.619961 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260621 05:10:45.620843 9733 tablet_copy_service.cc:140] P 08adad2fec7f4618bfbb0504365c5ecc: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer e6286f69e19545e883f56d392e7b8ea9 ({username='oryx', principal='oryx/127.3.131.188@KRBTEST.COM'} at 127.0.0.1:53628)
I20260621 05:10:45.620919 9733 tablet_copy_service.cc:161] P 08adad2fec7f4618bfbb0504365c5ecc: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer e6286f69e19545e883f56d392e7b8ea9 at {username='oryx', principal='oryx/127.3.131.188@KRBTEST.COM'} at 127.0.0.1:53628: session id = e6286f69e19545e883f56d392e7b8ea9-00000000000000000000000000000000
I20260621 05:10:45.621577 9733 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 08adad2fec7f4618bfbb0504365c5ecc: Tablet Copy: opened 0 blocks and 1 log segments
I20260621 05:10:45.622162 10365 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260621 05:10:45.623726 10365 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P e6286f69e19545e883f56d392e7b8ea9: tablet copy: Starting download of 0 data blocks...
I20260621 05:10:45.623901 10365 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P e6286f69e19545e883f56d392e7b8ea9: tablet copy: Starting download of 1 WAL segments...
I20260621 05:10:45.624823 10365 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P e6286f69e19545e883f56d392e7b8ea9: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260621 05:10:45.626058 9733 tablet_copy_service.cc:342] P 08adad2fec7f4618bfbb0504365c5ecc: Request end of tablet copy session e6286f69e19545e883f56d392e7b8ea9-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.3.131.188@KRBTEST.COM'} at 127.0.0.1:53628
I20260621 05:10:45.626116 9733 tablet_copy_service.cc:433] P 08adad2fec7f4618bfbb0504365c5ecc: ending tablet copy session e6286f69e19545e883f56d392e7b8ea9-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer e6286f69e19545e883f56d392e7b8ea9
I20260621 05:10:45.629189 10365 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 05:10:45.630658 10471 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:10:45.632367 10472 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:10:45.632603 10365 server_base.cc:1034] running on GCE node
W20260621 05:10:45.632721 10474 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:10:45.633033 10365 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 05:10:45.633176 10365 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 05:10:45.634298 10365 hybrid_clock.cc:648] HybridClock initialized: now 1782018645634291 us; error 28 us; skew 500 ppm
Jun 21 05:10:45 dist-test-slave-vrq1 krb5kdc[9668](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782018645, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.131.188@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 05:10:45.635939 10365 init.cc:377] Logged in from keytab as oryx/127.3.131.188@KRBTEST.COM (short username oryx)
I20260621 05:10:45.636224 10365 webserver.cc:460] Webserver started at http://127.3.131.188:34855/ using document root <none> and password file <none>
I20260621 05:10:45.636350 10365 fs_manager.cc:359] Metadata directory not provided
I20260621 05:10:45.636389 10365 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 05:10:45.637195 10365 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260621 05:10:45.637831 10481 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 05:10:45.638010 10365 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260621 05:10:45.638072 10365 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782018616648578-3598-0/minicluster-data/master-2/data,/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782018616648578-3598-0/minicluster-data/master-2/wal
uuid: "e6286f69e19545e883f56d392e7b8ea9"
format_stamp: "Formatted at 2026-06-21 05:10:45 on dist-test-slave-vrq1"
server_key: "e19903fad760d9372bbe20e226008456"
server_key_iv: "0365aaaa57e171e82aa778260566f896"
server_key_version: "encryptionkey@0"
I20260621 05:10:45.638137 10365 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782018616648578-3598-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782018616648578-3598-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782018616648578-3598-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
W20260621 05:10:45.670894 10458 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 05:10:45.671196 10458 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 05:10:45.671244 10458 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260621 05:10:45.671278 10365 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
W20260621 05:10:45.673561 10458 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
I20260621 05:10:45.673588 10365 kserver.cc:163] Server-wide thread pool size limit: 3276
W20260621 05:10:45.673641 10458 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 05:10:45.673696 10458 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.131.129
I20260621 05:10:45.676452 10458 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.131.148:35089
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782018616648578-3598-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782018616648578-3598-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782018616648578-3598-0/krb5kdc/oryx_127.3.131.129.keytab
--principal=oryx/127.3.131.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.131.129:38671
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782018616648578-3598-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.131.129
--webserver_port=37023
--tserver_master_addrs=127.3.131.190:33159,127.3.131.189:41383,127.3.131.188:32887
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10458
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.131.129
--log_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782018616648578-3598-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 21 Jun 2026 04:16:16 UTC on ccycloud.quasar-xycaha.root.comops.site
build id 4457881
I20260621 05:10:45.676826 10458 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 05:10:45.676870 10365 rpc_server.cc:307] RPC server started. Bound to: 127.3.131.188:32887
I20260621 05:10:45.677095 10458 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260621 05:10:45.677251 10365 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782018616648578-3598-0/minicluster-data/master-2/data/info.pb
I20260621 05:10:45.677600 10536 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.131.188:32887 every 8 connection(s)
I20260621 05:10:45.678395 10537 sys_catalog.cc:263] Verifying existing consensus state
I20260621 05:10:45.679005 10537 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e6286f69e19545e883f56d392e7b8ea9: Bootstrap starting.
I20260621 05:10:45.682358 10537 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e6286f69e19545e883f56d392e7b8ea9: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260621 05:10:45.682601 10537 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e6286f69e19545e883f56d392e7b8ea9: Bootstrap complete.
I20260621 05:10:45.682834 10537 raft_consensus.cc:348] T 00000000000000000000000000000000 P e6286f69e19545e883f56d392e7b8ea9 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "08adad2fec7f4618bfbb0504365c5ecc" member_type: VOTER last_known_addr { host: "127.3.131.190" port: 33159 } } peers { permanent_uuid: "65fa57b1227645fa8b95922a4da9e2ef" member_type: VOTER last_known_addr { host: "127.3.131.189" port: 41383 } attrs { promote: false } } peers { permanent_uuid: "e6286f69e19545e883f56d392e7b8ea9" member_type: NON_VOTER last_known_addr { host: "127.3.131.188" port: 32887 } attrs { promote: true } }
I20260621 05:10:45.682941 10537 raft_consensus.cc:729] T 00000000000000000000000000000000 P e6286f69e19545e883f56d392e7b8ea9 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: e6286f69e19545e883f56d392e7b8ea9, State: Initialized, Role: LEARNER
I20260621 05:10:45.682989 10537 consensus_queue.cc:260] T 00000000000000000000000000000000 P e6286f69e19545e883f56d392e7b8ea9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "08adad2fec7f4618bfbb0504365c5ecc" member_type: VOTER last_known_addr { host: "127.3.131.190" port: 33159 } } peers { permanent_uuid: "65fa57b1227645fa8b95922a4da9e2ef" member_type: VOTER last_known_addr { host: "127.3.131.189" port: 41383 } attrs { promote: false } } peers { permanent_uuid: "e6286f69e19545e883f56d392e7b8ea9" member_type: NON_VOTER last_known_addr { host: "127.3.131.188" port: 32887 } attrs { promote: true } }
I20260621 05:10:45.683223 10537 sys_catalog.cc:564] T 00000000000000000000000000000000 P e6286f69e19545e883f56d392e7b8ea9 [sys.catalog]: configured and running, proceeding with master startup.
I20260621 05:10:45.683228 10545 sys_catalog.cc:455] T 00000000000000000000000000000000 P e6286f69e19545e883f56d392e7b8ea9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "08adad2fec7f4618bfbb0504365c5ecc" member_type: VOTER last_known_addr { host: "127.3.131.190" port: 33159 } } peers { permanent_uuid: "65fa57b1227645fa8b95922a4da9e2ef" member_type: VOTER last_known_addr { host: "127.3.131.189" port: 41383 } attrs { promote: false } } peers { permanent_uuid: "e6286f69e19545e883f56d392e7b8ea9" member_type: NON_VOTER last_known_addr { host: "127.3.131.188" port: 32887 } attrs { promote: true } } }
I20260621 05:10:45.683481 10545 sys_catalog.cc:458] T 00000000000000000000000000000000 P e6286f69e19545e883f56d392e7b8ea9 [sys.catalog]: This master's current role is: LEARNER
W20260621 05:10:45.684018 10543 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:10:45.684058 10541 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:10:45.684656 10540 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:10:45.685356 10557 catalog_manager.cc:1259] Loaded cluster ID: 48235deea928424ca976ad57e26663f4
I20260621 05:10:45.685433 10557 catalog_manager.cc:1552] T 00000000000000000000000000000000 P e6286f69e19545e883f56d392e7b8ea9: loading cluster ID for follower catalog manager: success
I20260621 05:10:45.685559 10458 server_base.cc:1034] running on GCE node
I20260621 05:10:45.685775 10458 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 05:10:45.686031 10458 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 05:10:45.686439 10557 catalog_manager.cc:1574] T 00000000000000000000000000000000 P e6286f69e19545e883f56d392e7b8ea9: acquiring CA information for follower catalog manager: success
I20260621 05:10:45.686733 10557 catalog_manager.cc:1602] T 00000000000000000000000000000000 P e6286f69e19545e883f56d392e7b8ea9: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260621 05:10:45.687282 10458 hybrid_clock.cc:648] HybridClock initialized: now 1782018645687255 us; error 38 us; skew 500 ppm
Jun 21 05:10:45 dist-test-slave-vrq1 krb5kdc[9668](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782018645, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.131.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 05:10:45.690150 10458 init.cc:377] Logged in from keytab as oryx/127.3.131.129@KRBTEST.COM (short username oryx)
I20260621 05:10:45.690706 10458 webserver.cc:460] Webserver started at http://127.3.131.129:37023/ using document root <none> and password file <none>
I20260621 05:10:45.690984 10458 fs_manager.cc:359] Metadata directory not provided
I20260621 05:10:45.691074 10458 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 05:10:45.693114 10458 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260621 05:10:45.693882 10563 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 05:10:45.694144 10458 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260621 05:10:45.694267 10458 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782018616648578-3598-0/minicluster-data/ts-0/data,/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782018616648578-3598-0/minicluster-data/ts-0/wal
uuid: "e3bf3241a75e43fca924eac143beb63f"
format_stamp: "Formatted at 2026-06-21 05:10:44 on dist-test-slave-vrq1"
server_key: "a138dea9d5e4c03b21a7cc229054417a"
server_key_iv: "3772845953935a001ad243fcd48dbf99"
server_key_version: "encryptionkey@0"
I20260621 05:10:45.694628 10458 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782018616648578-3598-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782018616648578-3598-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782018616648578-3598-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 05:10:45.718351 10458 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 05:10:45.720911 10458 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 05:10:45.721045 10458 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 05:10:45.721349 10458 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 05:10:45.721684 10458 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 05:10:45.721719 10458 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 05:10:45.721742 10458 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 05:10:45.721755 10458 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 05:10:45.728101 10458 rpc_server.cc:307] RPC server started. Bound to: 127.3.131.129:38671
I20260621 05:10:45.728241 10676 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.131.129:38671 every 8 connection(s)
I20260621 05:10:45.728582 10458 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782018616648578-3598-0/minicluster-data/ts-0/data/info.pb
I20260621 05:10:45.734405 3598 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWZOx2_/build/release/bin/kudu as pid 10458
I20260621 05:10:45.734577 3598 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWZOx2_/build/release/bin/kudu with pid 9909
Jun 21 05:10:45 dist-test-slave-vrq1 krb5kdc[9668](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782018645, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.131.129@KRBTEST.COM for oryx/127.3.131.190@KRBTEST.COM
Jun 21 05:10:45 dist-test-slave-vrq1 krb5kdc[9668](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782018645, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.131.129@KRBTEST.COM for oryx/127.3.131.188@KRBTEST.COM
Jun 21 05:10:45 dist-test-slave-vrq1 krb5kdc[9668](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782018645, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.131.129@KRBTEST.COM for oryx/127.3.131.189@KRBTEST.COM
I20260621 05:10:45.741094 9752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:10:45.729771 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.131.129:45075 (local address 127.3.131.190:33159)
0621 05:10:45.729814 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0621 05:10:45.729816 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:10:45.729924 (+ 108us) server_negotiation.cc:407] Connection header received
0621 05:10:45.730324 (+ 400us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:10:45.730325 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:10:45.730338 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:10:45.730371 (+ 33us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:10:45.731230 (+ 859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:10:45.731728 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:10:45.733072 (+ 1344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:10:45.733256 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:10:45.733388 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:10:45.736462 (+ 3074us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:10:45.736463 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:10:45.736464 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:10:45.736474 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:10:45.739375 (+ 2901us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:10:45.739944 (+ 569us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:10:45.739945 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:10:45.739946 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:10:45.740000 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:10:45.740315 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:10:45.740316 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:10:45.740316 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:10:45.740471 (+ 155us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:10:45.740515 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0621 05:10:45.740921 (+ 406us) server_negotiation.cc:299] Negotiation successful
0621 05:10:45.741002 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260621 05:10:45.741153 10362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:10:45.729377 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.131.129:37677 (local address 127.3.131.189:41383)
0621 05:10:45.729426 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0621 05:10:45.729429 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0621 05:10:45.729890 (+ 461us) server_negotiation.cc:407] Connection header received
0621 05:10:45.730325 (+ 435us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:10:45.730326 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:10:45.730338 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:10:45.730371 (+ 33us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:10:45.731229 (+ 858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:10:45.731971 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:10:45.733020 (+ 1049us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:10:45.733314 (+ 294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:10:45.733376 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:10:45.737990 (+ 4614us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:10:45.737992 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:10:45.737992 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:10:45.738000 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:10:45.739738 (+ 1738us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:10:45.740169 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:10:45.740170 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:10:45.740171 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:10:45.740226 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:10:45.740420 (+ 194us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:10:45.740421 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:10:45.740422 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:10:45.740575 (+ 153us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:10:45.740617 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0621 05:10:45.740915 (+ 298us) server_negotiation.cc:299] Negotiation successful
0621 05:10:45.741009 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260621 05:10:45.741302 10681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:10:45.729389 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.131.188:32887 (local address 127.3.131.129:51165)
0621 05:10:45.729576 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0621 05:10:45.729614 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0621 05:10:45.730198 (+ 584us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:10:45.730472 (+ 274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:10:45.730476 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:10:45.730852 (+ 376us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:10:45.731081 (+ 229us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:10:45.731086 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:10:45.732273 (+ 1187us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:10:45.732274 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:10:45.732881 (+ 607us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:10:45.732883 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:10:45.733396 (+ 513us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:10:45.733397 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:10:45.733527 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:10:45.734367 (+ 840us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 05:10:45.734371 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:10:45.737536 (+ 3165us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 05:10:45.739719 (+ 2183us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:10:45.739719 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:10:45.739720 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:10:45.740154 (+ 434us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:10:45.740359 (+ 205us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:10:45.740360 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:10:45.740360 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:10:45.740408 (+ 48us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:10:45.740677 (+ 269us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 05:10:45.740677 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 05:10:45.740757 (+ 80us) client_negotiation.cc:769] Sending connection context
0621 05:10:45.740872 (+ 115us) client_negotiation.cc:240] Negotiation successful
0621 05:10:45.741053 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":130,"mutex_wait_us":17,"thread_start_us":92,"threads_started":1}
I20260621 05:10:45.741503 10684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:10:45.729389 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.131.189:41383 (local address 127.3.131.129:37677)
0621 05:10:45.729823 (+ 434us) negotiation.cc:107] Waiting for socket to connect
0621 05:10:45.729833 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0621 05:10:45.730225 (+ 392us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:10:45.730878 (+ 653us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:10:45.730879 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:10:45.731019 (+ 140us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:10:45.731083 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:10:45.731089 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:10:45.732064 (+ 975us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:10:45.732064 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:10:45.732947 (+ 883us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:10:45.732949 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:10:45.733395 (+ 446us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:10:45.733396 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:10:45.733529 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:10:45.734257 (+ 728us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 05:10:45.734262 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:10:45.737890 (+ 3628us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 05:10:45.739828 (+ 1938us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:10:45.739829 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:10:45.739829 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:10:45.740087 (+ 258us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:10:45.740296 (+ 209us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:10:45.740297 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:10:45.740297 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:10:45.740356 (+ 59us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:10:45.740661 (+ 305us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 05:10:45.740662 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 05:10:45.740758 (+ 96us) client_negotiation.cc:769] Sending connection context
0621 05:10:45.741229 (+ 471us) client_negotiation.cc:240] Negotiation successful
0621 05:10:45.741348 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":408,"mutex_wait_us":11,"spinlock_wait_cycles":768,"thread_start_us":98,"threads_started":1}
I20260621 05:10:45.741623 10682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:10:45.729440 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.131.129:51165 (local address 127.3.131.188:32887)
0621 05:10:45.729776 (+ 336us) server_negotiation.cc:206] Beginning negotiation
0621 05:10:45.729778 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:10:45.729790 (+ 12us) server_negotiation.cc:407] Connection header received
0621 05:10:45.730304 (+ 514us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:10:45.730305 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:10:45.730321 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:10:45.730397 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:10:45.731195 (+ 798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:10:45.732190 (+ 995us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:10:45.732968 (+ 778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:10:45.733270 (+ 302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:10:45.733333 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:10:45.737625 (+ 4292us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:10:45.737627 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:10:45.737627 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:10:45.737636 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:10:45.739634 (+ 1998us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:10:45.740228 (+ 594us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:10:45.740229 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:10:45.740230 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:10:45.740292 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:10:45.740469 (+ 177us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:10:45.740470 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:10:45.740470 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:10:45.740600 (+ 130us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:10:45.740654 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0621 05:10:45.741447 (+ 793us) server_negotiation.cc:299] Negotiation successful
0621 05:10:45.741541 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":305,"thread_start_us":62,"threads_started":1}
I20260621 05:10:45.742261 10679 heartbeater.cc:347] Connected to a master server at 127.3.131.189:41383
I20260621 05:10:45.742370 10679 heartbeater.cc:464] Registering TS with master...
I20260621 05:10:45.742529 10683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:10:45.729698 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.131.190:33159 (local address 127.3.131.129:45075)
0621 05:10:45.729862 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0621 05:10:45.729870 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0621 05:10:45.730223 (+ 353us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:10:45.730502 (+ 279us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:10:45.730503 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:10:45.730859 (+ 356us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:10:45.731083 (+ 224us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:10:45.731089 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:10:45.731828 (+ 739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:10:45.731828 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:10:45.732947 (+ 1119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:10:45.732949 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:10:45.733437 (+ 488us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:10:45.733438 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:10:45.733529 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:10:45.734247 (+ 718us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 05:10:45.734258 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:10:45.736346 (+ 2088us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 05:10:45.739472 (+ 3126us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:10:45.739481 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:10:45.739484 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:10:45.739847 (+ 363us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:10:45.740092 (+ 245us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:10:45.740093 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:10:45.740093 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:10:45.740235 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:10:45.740575 (+ 340us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 05:10:45.740582 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 05:10:45.740757 (+ 175us) client_negotiation.cc:769] Sending connection context
0621 05:10:45.742246 (+ 1489us) client_negotiation.cc:240] Negotiation successful
0621 05:10:45.742357 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":143,"mutex_wait_us":5,"thread_start_us":64,"threads_started":1}
I20260621 05:10:45.742615 10679 heartbeater.cc:511] Master 127.3.131.189:41383 requested a full tablet report, sending...
I20260621 05:10:45.742786 3598 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWZOx2_/build/release/bin/kudu
/tmp/dist-test-taskWZOx2_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782018616648578-3598-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782018616648578-3598-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782018616648578-3598-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782018616648578-3598-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.131.130:34425
--local_ip_for_outbound_sockets=127.3.131.130
--tserver_master_addrs=127.3.131.190:33159
--webserver_port=37661
--webserver_interface=127.3.131.130
--keytab_file=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782018616648578-3598-0/krb5kdc/oryx_127.3.131.130.keytab
--principal=oryx/127.3.131.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.131.148:35089
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.3.131.190:33159,127.3.131.189:41383
--tserver_master_addrs=127.3.131.190:33159,127.3.131.189:41383,127.3.131.188:32887 with env {KRB5CCNAME=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782018616648578-3598-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782018616648578-3598-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782018616648578-3598-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260621 05:10:45.743098 10677 heartbeater.cc:347] Connected to a master server at 127.3.131.188:32887
I20260621 05:10:45.743151 10677 heartbeater.cc:464] Registering TS with master...
I20260621 05:10:45.743243 10677 heartbeater.cc:511] Master 127.3.131.188:32887 requested a full tablet report, sending...
I20260621 05:10:45.743250 10311 ts_manager.cc:194] Registered new tserver with Master: e3bf3241a75e43fca924eac143beb63f (127.3.131.129:38671)
I20260621 05:10:45.743517 10678 heartbeater.cc:347] Connected to a master server at 127.3.131.190:33159
I20260621 05:10:45.743571 10678 heartbeater.cc:464] Registering TS with master...
I20260621 05:10:45.743619 10493 ts_manager.cc:194] Registered new tserver with Master: e3bf3241a75e43fca924eac143beb63f (127.3.131.129:38671)
I20260621 05:10:45.743637 10678 heartbeater.cc:511] Master 127.3.131.190:33159 requested a full tablet report, sending...
I20260621 05:10:45.743919 9713 ts_manager.cc:194] Re-registered known tserver with Master: e3bf3241a75e43fca924eac143beb63f (127.3.131.129:38671)
I20260621 05:10:45.744436 9713 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.3.131.129@KRBTEST.COM'} at 127.3.131.129:45075
W20260621 05:10:45.827705 10685 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 05:10:45.827912 10685 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 05:10:45.827936 10685 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 05:10:45.829037 10685 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 05:10:45.829072 10685 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 05:10:45.829097 10685 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.131.130
I20260621 05:10:45.830482 10685 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.131.148:35089
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782018616648578-3598-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782018616648578-3598-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782018616648578-3598-0/krb5kdc/oryx_127.3.131.130.keytab
--principal=oryx/127.3.131.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.131.130:34425
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782018616648578-3598-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.131.130
--webserver_port=37661
--tserver_master_addrs=127.3.131.190:33159,127.3.131.189:41383,127.3.131.188:32887
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10685
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.131.130
--log_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782018616648578-3598-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 21 Jun 2026 04:16:16 UTC on ccycloud.quasar-xycaha.root.comops.site
build id 4457881
I20260621 05:10:45.830663 10685 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 05:10:45.830842 10685 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 05:10:45.835947 10690 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:10:45.835937 10693 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:10:45.836128 10685 server_base.cc:1034] running on GCE node
W20260621 05:10:45.835935 10691 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:10:45.836406 10685 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 05:10:45.836675 10685 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 05:10:45.837884 10685 hybrid_clock.cc:648] HybridClock initialized: now 1782018645837864 us; error 31 us; skew 500 ppm
Jun 21 05:10:45 dist-test-slave-vrq1 krb5kdc[9668](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782018645, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.131.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 05:10:45.840874 10685 init.cc:377] Logged in from keytab as oryx/127.3.131.130@KRBTEST.COM (short username oryx)
I20260621 05:10:45.841373 10685 webserver.cc:460] Webserver started at http://127.3.131.130:37661/ using document root <none> and password file <none>
I20260621 05:10:45.841639 10685 fs_manager.cc:359] Metadata directory not provided
I20260621 05:10:45.841738 10685 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 05:10:45.843394 10685 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260621 05:10:45.844132 10700 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 05:10:45.844309 10685 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260621 05:10:45.844414 10685 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782018616648578-3598-0/minicluster-data/ts-1/data,/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782018616648578-3598-0/minicluster-data/ts-1/wal
uuid: "8243bd6da6a04cdb80f9f0c681600639"
format_stamp: "Formatted at 2026-06-21 05:10:44 on dist-test-slave-vrq1"
server_key: "47d063eac3a7917bf4aa6b3561563013"
server_key_iv: "db8daa4d92d8eecc16660176762271b7"
server_key_version: "encryptionkey@0"
I20260621 05:10:45.844818 10685 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782018616648578-3598-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782018616648578-3598-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782018616648578-3598-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 05:10:45.854593 10685 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 05:10:45.856918 10685 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 05:10:45.857098 10685 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 05:10:45.857362 10685 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 05:10:45.857689 10685 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 05:10:45.857748 10685 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 05:10:45.857795 10685 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 05:10:45.857827 10685 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 05:10:45.864507 10685 rpc_server.cc:307] RPC server started. Bound to: 127.3.131.130:34425
I20260621 05:10:45.864553 10813 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.131.130:34425 every 8 connection(s)
I20260621 05:10:45.864945 10685 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782018616648578-3598-0/minicluster-data/ts-1/data/info.pb
I20260621 05:10:45.868396 3598 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWZOx2_/build/release/bin/kudu as pid 10685
I20260621 05:10:45.868564 3598 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWZOx2_/build/release/bin/kudu with pid 10043
Jun 21 05:10:45 dist-test-slave-vrq1 krb5kdc[9668](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782018645, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.131.130@KRBTEST.COM for oryx/127.3.131.188@KRBTEST.COM
Jun 21 05:10:45 dist-test-slave-vrq1 krb5kdc[9668](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782018645, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.131.130@KRBTEST.COM for oryx/127.3.131.189@KRBTEST.COM
I20260621 05:10:45.875152 3598 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWZOx2_/build/release/bin/kudu
/tmp/dist-test-taskWZOx2_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782018616648578-3598-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782018616648578-3598-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782018616648578-3598-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782018616648578-3598-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.131.131:37567
--local_ip_for_outbound_sockets=127.3.131.131
--tserver_master_addrs=127.3.131.190:33159
--webserver_port=44923
--webserver_interface=127.3.131.131
--keytab_file=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782018616648578-3598-0/krb5kdc/oryx_127.3.131.131.keytab
--principal=oryx/127.3.131.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.131.148:35089
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.3.131.190:33159,127.3.131.189:41383
--tserver_master_addrs=127.3.131.190:33159,127.3.131.189:41383,127.3.131.188:32887 with env {KRB5CCNAME=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782018616648578-3598-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782018616648578-3598-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782018616648578-3598-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260621 05:10:45.877974 10682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:10:45.866533 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.131.130:39741 (local address 127.3.131.188:32887)
0621 05:10:45.866608 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0621 05:10:45.866610 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:10:45.866664 (+ 54us) server_negotiation.cc:407] Connection header received
0621 05:10:45.867623 (+ 959us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:10:45.867624 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:10:45.867640 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:10:45.867673 (+ 33us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:10:45.869322 (+ 1649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:10:45.869842 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:10:45.870905 (+ 1063us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:10:45.871119 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:10:45.871222 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:10:45.874220 (+ 2998us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:10:45.874222 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:10:45.874222 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:10:45.874231 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:10:45.876137 (+ 1906us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:10:45.876739 (+ 602us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:10:45.876741 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:10:45.876741 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:10:45.876804 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:10:45.877145 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:10:45.877146 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:10:45.877147 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:10:45.877314 (+ 167us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:10:45.877404 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0621 05:10:45.877773 (+ 369us) server_negotiation.cc:299] Negotiation successful
0621 05:10:45.877873 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260621 05:10:45.878687 10362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:10:45.865914 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.131.130:50161 (local address 127.3.131.189:41383)
0621 05:10:45.866038 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0621 05:10:45.866041 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0621 05:10:45.866260 (+ 219us) server_negotiation.cc:407] Connection header received
0621 05:10:45.867753 (+ 1493us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:10:45.867755 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:10:45.867766 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:10:45.867823 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:10:45.869437 (+ 1614us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:10:45.870160 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:10:45.871335 (+ 1175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:10:45.871644 (+ 309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:10:45.871726 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:10:45.874923 (+ 3197us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:10:45.874924 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:10:45.874925 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:10:45.874933 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:10:45.877468 (+ 2535us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:10:45.877802 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:10:45.877803 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:10:45.877803 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:10:45.877863 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:10:45.878061 (+ 198us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:10:45.878062 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:10:45.878063 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:10:45.878201 (+ 138us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:10:45.878243 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0621 05:10:45.878444 (+ 201us) server_negotiation.cc:299] Negotiation successful
0621 05:10:45.878533 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91}
I20260621 05:10:45.879004 10820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:10:45.866477 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.131.188:32887 (local address 127.3.131.130:39741)
0621 05:10:45.866613 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0621 05:10:45.866622 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0621 05:10:45.867478 (+ 856us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:10:45.867775 (+ 297us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:10:45.867776 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:10:45.868343 (+ 567us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:10:45.869229 (+ 886us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:10:45.869236 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:10:45.869956 (+ 720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:10:45.869957 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:10:45.870811 (+ 854us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:10:45.870813 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:10:45.871224 (+ 411us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:10:45.871225 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:10:45.871384 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:10:45.872015 (+ 631us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 05:10:45.872041 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:10:45.874039 (+ 1998us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 05:10:45.876262 (+ 2223us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:10:45.876271 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:10:45.876273 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:10:45.876653 (+ 380us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:10:45.876875 (+ 222us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:10:45.876876 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:10:45.876876 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:10:45.877020 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:10:45.877413 (+ 393us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 05:10:45.877420 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 05:10:45.877601 (+ 181us) client_negotiation.cc:769] Sending connection context
0621 05:10:45.878333 (+ 732us) client_negotiation.cc:240] Negotiation successful
0621 05:10:45.878519 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119,"mutex_wait_us":568,"thread_start_us":88,"threads_started":1}
I20260621 05:10:45.879136 10818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:10:45.865812 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.131.189:41383 (local address 127.3.131.130:50161)
0621 05:10:45.866120 (+ 308us) negotiation.cc:107] Waiting for socket to connect
0621 05:10:45.866165 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0621 05:10:45.867447 (+ 1282us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:10:45.867843 (+ 396us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:10:45.867844 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:10:45.868342 (+ 498us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:10:45.869332 (+ 990us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:10:45.869335 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:10:45.870349 (+ 1014us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:10:45.870350 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:10:45.871202 (+ 852us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:10:45.871205 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:10:45.871724 (+ 519us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:10:45.871725 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:10:45.871797 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:10:45.872160 (+ 363us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 05:10:45.872165 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:10:45.874819 (+ 2654us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 05:10:45.877549 (+ 2730us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:10:45.877550 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:10:45.877551 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:10:45.877718 (+ 167us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:10:45.877936 (+ 218us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:10:45.877937 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:10:45.877937 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:10:45.877996 (+ 59us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:10:45.878291 (+ 295us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 05:10:45.878292 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 05:10:45.878335 (+ 43us) client_negotiation.cc:769] Sending connection context
0621 05:10:45.878808 (+ 473us) client_negotiation.cc:240] Negotiation successful
0621 05:10:45.878921 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":209,"mutex_wait_us":28,"thread_start_us":134,"threads_started":1}
I20260621 05:10:45.879787 10816 heartbeater.cc:347] Connected to a master server at 127.3.131.189:41383
I20260621 05:10:45.879807 10814 heartbeater.cc:347] Connected to a master server at 127.3.131.188:32887
I20260621 05:10:45.879935 10814 heartbeater.cc:464] Registering TS with master...
I20260621 05:10:45.880174 10814 heartbeater.cc:511] Master 127.3.131.188:32887 requested a full tablet report, sending...
I20260621 05:10:45.879935 10816 heartbeater.cc:464] Registering TS with master...
I20260621 05:10:45.880446 10816 heartbeater.cc:511] Master 127.3.131.189:41383 requested a full tablet report, sending...
I20260621 05:10:45.880707 10493 ts_manager.cc:194] Registered new tserver with Master: 8243bd6da6a04cdb80f9f0c681600639 (127.3.131.130:34425)
I20260621 05:10:45.881064 10311 ts_manager.cc:194] Registered new tserver with Master: 8243bd6da6a04cdb80f9f0c681600639 (127.3.131.130:34425)
Jun 21 05:10:45 dist-test-slave-vrq1 krb5kdc[9668](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782018645, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.131.130@KRBTEST.COM for oryx/127.3.131.190@KRBTEST.COM
I20260621 05:10:45.884850 9752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:10:45.866346 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.131.130:37095 (local address 127.3.131.190:33159)
0621 05:10:45.866421 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0621 05:10:45.866424 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0621 05:10:45.866486 (+ 62us) server_negotiation.cc:407] Connection header received
0621 05:10:45.867582 (+ 1096us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:10:45.867584 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:10:45.867626 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:10:45.867694 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:10:45.875371 (+ 7677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:10:45.876225 (+ 854us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:10:45.878394 (+ 2169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:10:45.878678 (+ 284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:10:45.878724 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:10:45.882003 (+ 3279us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:10:45.882004 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:10:45.882005 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:10:45.882013 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:10:45.883635 (+ 1622us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:10:45.884010 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:10:45.884011 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:10:45.884012 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:10:45.884074 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:10:45.884260 (+ 186us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:10:45.884261 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:10:45.884261 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:10:45.884417 (+ 156us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:10:45.884497 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0621 05:10:45.884654 (+ 157us) server_negotiation.cc:299] Negotiation successful
0621 05:10:45.884729 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260621 05:10:45.884864 10819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:10:45.866248 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.131.190:33159 (local address 127.3.131.130:37095)
0621 05:10:45.866424 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0621 05:10:45.866434 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0621 05:10:45.867447 (+ 1013us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:10:45.867767 (+ 320us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:10:45.867771 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:10:45.868342 (+ 571us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:10:45.875243 (+ 6901us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:10:45.875247 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:10:45.876671 (+ 1424us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:10:45.876671 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:10:45.878292 (+ 1621us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:10:45.878294 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:10:45.878814 (+ 520us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:10:45.878816 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:10:45.878880 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:10:45.879724 (+ 844us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 05:10:45.879729 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:10:45.881899 (+ 2170us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 05:10:45.883750 (+ 1851us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:10:45.883751 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:10:45.883751 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:10:45.883928 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:10:45.884125 (+ 197us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:10:45.884125 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:10:45.884126 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:10:45.884181 (+ 55us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:10:45.884487 (+ 306us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 05:10:45.884488 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 05:10:45.884532 (+ 44us) client_negotiation.cc:769] Sending connection context
0621 05:10:45.884611 (+ 79us) client_negotiation.cc:240] Negotiation successful
0621 05:10:45.884726 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":151,"mutex_wait_us":24,"thread_start_us":71,"threads_started":1}
I20260621 05:10:45.885185 10815 heartbeater.cc:347] Connected to a master server at 127.3.131.190:33159
I20260621 05:10:45.885246 10815 heartbeater.cc:464] Registering TS with master...
I20260621 05:10:45.885324 10815 heartbeater.cc:511] Master 127.3.131.190:33159 requested a full tablet report, sending...
I20260621 05:10:45.885560 9713 ts_manager.cc:194] Re-registered known tserver with Master: 8243bd6da6a04cdb80f9f0c681600639 (127.3.131.130:34425)
I20260621 05:10:45.885938 9713 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.3.131.130@KRBTEST.COM'} at 127.3.131.130:37095
W20260621 05:10:45.962795 10821 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 05:10:45.963014 10821 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 05:10:45.963042 10821 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 05:10:45.964336 10821 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 05:10:45.964387 10821 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 05:10:45.964417 10821 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.131.131
I20260621 05:10:45.965991 10821 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.131.148:35089
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782018616648578-3598-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782018616648578-3598-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782018616648578-3598-0/krb5kdc/oryx_127.3.131.131.keytab
--principal=oryx/127.3.131.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.131.131:37567
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782018616648578-3598-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.131.131
--webserver_port=44923
--tserver_master_addrs=127.3.131.190:33159,127.3.131.189:41383,127.3.131.188:32887
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10821
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.131.131
--log_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782018616648578-3598-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 21 Jun 2026 04:16:16 UTC on ccycloud.quasar-xycaha.root.comops.site
build id 4457881
I20260621 05:10:45.966270 10821 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 05:10:45.966526 10821 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 05:10:45.971727 10829 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:10:45.971737 10826 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:10:45.971887 10821 server_base.cc:1034] running on GCE node
W20260621 05:10:45.971762 10827 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:10:45.972347 10821 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 05:10:45.972584 10821 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 05:10:45.973753 10821 hybrid_clock.cc:648] HybridClock initialized: now 1782018645973737 us; error 30 us; skew 500 ppm
Jun 21 05:10:45 dist-test-slave-vrq1 krb5kdc[9668](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782018645, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.131.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 05:10:45.976765 10821 init.cc:377] Logged in from keytab as oryx/127.3.131.131@KRBTEST.COM (short username oryx)
I20260621 05:10:45.977255 10821 webserver.cc:460] Webserver started at http://127.3.131.131:44923/ using document root <none> and password file <none>
I20260621 05:10:45.977523 10821 fs_manager.cc:359] Metadata directory not provided
I20260621 05:10:45.977617 10821 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 05:10:45.979364 10821 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260621 05:10:45.980165 10836 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 05:10:45.980456 10821 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260621 05:10:45.980588 10821 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782018616648578-3598-0/minicluster-data/ts-2/data,/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782018616648578-3598-0/minicluster-data/ts-2/wal
uuid: "7e1eb017cbe947a08ad9b9e092651410"
format_stamp: "Formatted at 2026-06-21 05:10:44 on dist-test-slave-vrq1"
server_key: "65010e9c28dcf53ff9c5cf203e36540c"
server_key_iv: "5888a21b0180e69ce32f332b500ce118"
server_key_version: "encryptionkey@0"
I20260621 05:10:45.980973 10821 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782018616648578-3598-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782018616648578-3598-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782018616648578-3598-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 05:10:45.993520 10821 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 05:10:45.996084 10821 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 05:10:45.996274 10821 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 05:10:45.996524 10821 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 05:10:45.996940 10821 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 05:10:45.997000 10821 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 05:10:45.997044 10821 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 05:10:45.997071 10821 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 05:10:46.003348 10821 rpc_server.cc:307] RPC server started. Bound to: 127.3.131.131:37567
I20260621 05:10:46.003604 10949 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.131.131:37567 every 8 connection(s)
I20260621 05:10:46.003867 10821 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782018616648578-3598-0/minicluster-data/ts-2/data/info.pb
Jun 21 05:10:46 dist-test-slave-vrq1 krb5kdc[9668](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782018645, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.131.131@KRBTEST.COM for oryx/127.3.131.190@KRBTEST.COM
I20260621 05:10:46.011901 3598 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWZOx2_/build/release/bin/kudu as pid 10821
Jun 21 05:10:46 dist-test-slave-vrq1 krb5kdc[9668](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782018645, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.131.131@KRBTEST.COM for oryx/127.3.131.189@KRBTEST.COM
Jun 21 05:10:46 dist-test-slave-vrq1 krb5kdc[9668](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782018645, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.131.131@KRBTEST.COM for oryx/127.3.131.188@KRBTEST.COM
I20260621 05:10:46.016754 10956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:10:46.005225 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.131.190:33159 (local address 127.3.131.131:49123)
0621 05:10:46.005828 (+ 603us) negotiation.cc:107] Waiting for socket to connect
0621 05:10:46.005837 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0621 05:10:46.005888 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:10:46.006077 (+ 189us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:10:46.006078 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:10:46.006495 (+ 417us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:10:46.006737 (+ 242us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:10:46.006743 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:10:46.007586 (+ 843us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:10:46.007587 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:10:46.008789 (+ 1202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:10:46.008791 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:10:46.009233 (+ 442us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:10:46.009234 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:10:46.009386 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:10:46.010164 (+ 778us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 05:10:46.010169 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:10:46.012153 (+ 1984us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 05:10:46.014258 (+ 2105us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:10:46.014268 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:10:46.014271 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:10:46.014600 (+ 329us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:10:46.015046 (+ 446us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:10:46.015047 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:10:46.015047 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:10:46.015226 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:10:46.015922 (+ 696us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 05:10:46.015930 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 05:10:46.016137 (+ 207us) client_negotiation.cc:769] Sending connection context
0621 05:10:46.016300 (+ 163us) client_negotiation.cc:240] Negotiation successful
0621 05:10:46.016493 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":587,"mutex_wait_us":42,"thread_start_us":64,"threads_started":1}
I20260621 05:10:46.017374 9752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:10:46.005294 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.131.131:49123 (local address 127.3.131.190:33159)
0621 05:10:46.005338 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0621 05:10:46.005340 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:10:46.005912 (+ 572us) server_negotiation.cc:407] Connection header received
0621 05:10:46.005943 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:10:46.005945 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:10:46.005957 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:10:46.006010 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:10:46.006860 (+ 850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:10:46.007474 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:10:46.008880 (+ 1406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:10:46.009114 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:10:46.009170 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:10:46.012283 (+ 3113us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:10:46.012285 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:10:46.012286 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:10:46.012294 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:10:46.014123 (+ 1829us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:10:46.014665 (+ 542us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:10:46.014666 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:10:46.014667 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:10:46.014731 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:10:46.015648 (+ 917us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:10:46.015649 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:10:46.015650 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:10:46.015821 (+ 171us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:10:46.015891 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0621 05:10:46.017194 (+ 1303us) server_negotiation.cc:299] Negotiation successful
0621 05:10:46.017281 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260621 05:10:46.017730 10951 heartbeater.cc:347] Connected to a master server at 127.3.131.190:33159
I20260621 05:10:46.017846 10951 heartbeater.cc:464] Registering TS with master...
I20260621 05:10:46.018069 10951 heartbeater.cc:511] Master 127.3.131.190:33159 requested a full tablet report, sending...
I20260621 05:10:46.018364 10682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:10:46.004737 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.131.131:40795 (local address 127.3.131.188:32887)
0621 05:10:46.004804 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0621 05:10:46.004806 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:10:46.005046 (+ 240us) server_negotiation.cc:407] Connection header received
0621 05:10:46.005936 (+ 890us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:10:46.005937 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:10:46.005949 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:10:46.006074 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:10:46.007905 (+ 1831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:10:46.008644 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:10:46.009585 (+ 941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:10:46.009865 (+ 280us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:10:46.009930 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:10:46.014267 (+ 4337us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:10:46.014268 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:10:46.014269 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:10:46.014275 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:10:46.016894 (+ 2619us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:10:46.017267 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:10:46.017268 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:10:46.017269 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:10:46.017333 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:10:46.017543 (+ 210us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:10:46.017544 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:10:46.017545 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:10:46.017686 (+ 141us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:10:46.017749 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0621 05:10:46.018165 (+ 416us) server_negotiation.cc:299] Negotiation successful
0621 05:10:46.018257 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260621 05:10:46.018090 10954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:10:46.004740 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.131.188:32887 (local address 127.3.131.131:40795)
0621 05:10:46.004913 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0621 05:10:46.004951 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0621 05:10:46.005644 (+ 693us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:10:46.006109 (+ 465us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:10:46.006110 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:10:46.006499 (+ 389us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:10:46.006738 (+ 239us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:10:46.006745 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:10:46.008726 (+ 1981us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:10:46.008727 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:10:46.009502 (+ 775us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:10:46.009504 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:10:46.009962 (+ 458us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:10:46.009964 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:10:46.010055 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:10:46.010428 (+ 373us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 05:10:46.010433 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:10:46.014190 (+ 3757us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 05:10:46.016988 (+ 2798us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:10:46.016988 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:10:46.016989 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:10:46.017187 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:10:46.017401 (+ 214us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:10:46.017401 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:10:46.017402 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:10:46.017460 (+ 58us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:10:46.017776 (+ 316us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 05:10:46.017776 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 05:10:46.017821 (+ 45us) client_negotiation.cc:769] Sending connection context
0621 05:10:46.017887 (+ 66us) client_negotiation.cc:240] Negotiation successful
0621 05:10:46.017987 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":131,"mutex_wait_us":23,"thread_start_us":83,"threads_started":1}
I20260621 05:10:46.018688 10362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:10:46.005130 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.131.131:51533 (local address 127.3.131.189:41383)
0621 05:10:46.005184 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0621 05:10:46.005186 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:10:46.005254 (+ 68us) server_negotiation.cc:407] Connection header received
0621 05:10:46.005767 (+ 513us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:10:46.005769 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:10:46.005785 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:10:46.005843 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:10:46.007257 (+ 1414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:10:46.007794 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:10:46.008944 (+ 1150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:10:46.009243 (+ 299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:10:46.009310 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:10:46.012925 (+ 3615us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:10:46.012927 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:10:46.012927 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:10:46.012936 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:10:46.015937 (+ 3001us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:10:46.017762 (+ 1825us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:10:46.017764 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:10:46.017764 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:10:46.017830 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:10:46.018090 (+ 260us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:10:46.018091 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:10:46.018091 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:10:46.018214 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:10:46.018268 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0621 05:10:46.018509 (+ 241us) server_negotiation.cc:299] Negotiation successful
0621 05:10:46.018589 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260621 05:10:46.018744 9713 ts_manager.cc:194] Re-registered known tserver with Master: 7e1eb017cbe947a08ad9b9e092651410 (127.3.131.131:37567)
I20260621 05:10:46.019052 10955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:10:46.005054 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.131.189:41383 (local address 127.3.131.131:51533)
0621 05:10:46.005189 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0621 05:10:46.005199 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0621 05:10:46.005653 (+ 454us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:10:46.005929 (+ 276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:10:46.005933 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:10:46.006499 (+ 566us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:10:46.006738 (+ 239us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:10:46.006745 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:10:46.007881 (+ 1136us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:10:46.007882 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:10:46.008805 (+ 923us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:10:46.008808 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:10:46.009304 (+ 496us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:10:46.009305 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:10:46.009386 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:10:46.010115 (+ 729us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 05:10:46.010127 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:10:46.012787 (+ 2660us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 05:10:46.016037 (+ 3250us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:10:46.016038 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:10:46.016039 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:10:46.017687 (+ 1648us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:10:46.017939 (+ 252us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:10:46.017940 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:10:46.017940 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:10:46.018014 (+ 74us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:10:46.018340 (+ 326us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 05:10:46.018341 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 05:10:46.018383 (+ 42us) client_negotiation.cc:769] Sending connection context
0621 05:10:46.018813 (+ 430us) client_negotiation.cc:240] Negotiation successful
0621 05:10:46.018929 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":112,"mutex_wait_us":5,"thread_start_us":74,"threads_started":1}
I20260621 05:10:46.019235 9713 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.3.131.131@KRBTEST.COM'} at 127.3.131.131:49123
I20260621 05:10:46.019387 10952 heartbeater.cc:347] Connected to a master server at 127.3.131.189:41383
I20260621 05:10:46.019439 10952 heartbeater.cc:464] Registering TS with master...
I20260621 05:10:46.019517 10952 heartbeater.cc:511] Master 127.3.131.189:41383 requested a full tablet report, sending...
I20260621 05:10:46.019665 10950 heartbeater.cc:347] Connected to a master server at 127.3.131.188:32887
I20260621 05:10:46.019701 10950 heartbeater.cc:464] Registering TS with master...
I20260621 05:10:46.019762 10950 heartbeater.cc:511] Master 127.3.131.188:32887 requested a full tablet report, sending...
I20260621 05:10:46.020663 10493 ts_manager.cc:194] Registered new tserver with Master: 7e1eb017cbe947a08ad9b9e092651410 (127.3.131.131:37567)
I20260621 05:10:46.020675 10311 ts_manager.cc:194] Registered new tserver with Master: 7e1eb017cbe947a08ad9b9e092651410 (127.3.131.131:37567)
I20260621 05:10:46.023553 10964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:10:46.013740 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35238 (local address 127.3.131.189:41383)
0621 05:10:46.014312 (+ 572us) server_negotiation.cc:206] Beginning negotiation
0621 05:10:46.014314 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:10:46.014419 (+ 105us) server_negotiation.cc:407] Connection header received
0621 05:10:46.014524 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:10:46.014525 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:10:46.014538 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:10:46.014607 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:10:46.015552 (+ 945us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:10:46.016358 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:10:46.017616 (+ 1258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:10:46.017998 (+ 382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:10:46.018060 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:10:46.020384 (+ 2324us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:10:46.020385 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:10:46.020386 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:10:46.020393 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:10:46.022154 (+ 1761us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:10:46.022540 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:10:46.022541 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:10:46.022542 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:10:46.022606 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:10:46.022842 (+ 236us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:10:46.022843 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:10:46.022843 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:10:46.022957 (+ 114us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:10:46.023062 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0621 05:10:46.023250 (+ 188us) server_negotiation.cc:299] Negotiation successful
0621 05:10:46.023418 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":549,"thread_start_us":69,"threads_started":1}
I20260621 05:10:46.024272 10963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:10:46.013565 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53638 (local address 127.3.131.190:33159)
0621 05:10:46.013859 (+ 294us) server_negotiation.cc:206] Beginning negotiation
0621 05:10:46.013862 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0621 05:10:46.014294 (+ 432us) server_negotiation.cc:407] Connection header received
0621 05:10:46.014845 (+ 551us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:10:46.014846 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:10:46.014861 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:10:46.014910 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:10:46.016414 (+ 1504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:10:46.017229 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:10:46.019604 (+ 2375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:10:46.019904 (+ 300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:10:46.019945 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:10:46.021313 (+ 1368us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:10:46.021314 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:10:46.021314 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:10:46.021323 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:10:46.023059 (+ 1736us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:10:46.023428 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:10:46.023430 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:10:46.023430 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:10:46.023486 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:10:46.023713 (+ 227us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:10:46.023714 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:10:46.023714 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:10:46.023834 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:10:46.023908 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0621 05:10:46.024082 (+ 174us) server_negotiation.cc:299] Negotiation successful
0621 05:10:46.024167 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":103,"thread_start_us":79,"threads_started":1}
I20260621 05:10:46.028103 10965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:10:46.014082 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35454 (local address 127.3.131.188:32887)
0621 05:10:46.014212 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0621 05:10:46.014215 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0621 05:10:46.020400 (+ 6185us) server_negotiation.cc:407] Connection header received
0621 05:10:46.020449 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:10:46.020450 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:10:46.020464 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:10:46.021363 (+ 899us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:10:46.021970 (+ 607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:10:46.022736 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:10:46.023771 (+ 1035us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:10:46.024051 (+ 280us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:10:46.024132 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:10:46.025427 (+ 1295us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:10:46.025428 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:10:46.025429 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:10:46.025435 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:10:46.026868 (+ 1433us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:10:46.027279 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:10:46.027280 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:10:46.027280 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:10:46.027346 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:10:46.027587 (+ 241us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:10:46.027588 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:10:46.027589 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:10:46.027718 (+ 129us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:10:46.027804 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0621 05:10:46.027921 (+ 117us) server_negotiation.cc:299] Negotiation successful
0621 05:10:46.028005 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":109,"thread_start_us":86,"threads_started":1}
I20260621 05:10:46.026273 9713 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53638:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260621 05:10:46.028384 9713 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260621 05:10:46.036471 10271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:10:46.032180 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.131.131:37567 (local address 127.0.0.1:44338)
0621 05:10:46.032213 (+ 33us) negotiation.cc:107] Waiting for socket to connect
0621 05:10:46.032224 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0621 05:10:46.032266 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:10:46.033034 (+ 768us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:10:46.033035 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:10:46.033037 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 05:10:46.033079 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:10:46.033080 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:10:46.034248 (+ 1168us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:10:46.034249 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:10:46.035500 (+ 1251us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:10:46.035503 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:10:46.036254 (+ 751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:10:46.036254 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:10:46.036310 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:10:46.036313 (+ 3us) client_negotiation.cc:769] Sending connection context
0621 05:10:46.036359 (+ 46us) client_negotiation.cc:240] Negotiation successful
0621 05:10:46.036397 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":18,"mutex_wait_us":101}
I20260621 05:10:46.036481 10972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:10:46.032332 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44338 (local address 127.3.131.131:37567)
0621 05:10:46.032849 (+ 517us) server_negotiation.cc:206] Beginning negotiation
0621 05:10:46.032854 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 05:10:46.032865 (+ 11us) server_negotiation.cc:407] Connection header received
0621 05:10:46.032909 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:10:46.032911 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:10:46.032930 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:10:46.033150 (+ 220us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 05:10:46.033224 (+ 74us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:10:46.034159 (+ 935us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:10:46.035590 (+ 1431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:10:46.036155 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:10:46.036252 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:10:46.036338 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0621 05:10:46.036371 (+ 33us) server_negotiation.cc:299] Negotiation successful
0621 05:10:46.036410 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":478,"thread_start_us":94,"threads_started":1}
I20260621 05:10:46.036947 10973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:10:46.032358 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56808 (local address 127.3.131.130:34425)
0621 05:10:46.032523 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0621 05:10:46.032528 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 05:10:46.032974 (+ 446us) server_negotiation.cc:407] Connection header received
0621 05:10:46.033098 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:10:46.033101 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:10:46.033117 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:10:46.033208 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 05:10:46.033314 (+ 106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:10:46.034394 (+ 1080us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:10:46.035735 (+ 1341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:10:46.036458 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:10:46.036519 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:10:46.036586 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0621 05:10:46.036804 (+ 218us) server_negotiation.cc:299] Negotiation successful
0621 05:10:46.036849 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":129,"thread_start_us":95,"threads_started":1}
I20260621 05:10:46.037143 10974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:10:46.032522 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56678 (local address 127.3.131.129:38671)
0621 05:10:46.032699 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0621 05:10:46.032703 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 05:10:46.032712 (+ 9us) server_negotiation.cc:407] Connection header received
0621 05:10:46.032753 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:10:46.032755 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:10:46.032770 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:10:46.032856 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 05:10:46.032965 (+ 109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:10:46.034421 (+ 1456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:10:46.035545 (+ 1124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:10:46.036284 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:10:46.036357 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:10:46.036424 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0621 05:10:46.036945 (+ 521us) server_negotiation.cc:299] Negotiation successful
0621 05:10:46.037066 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":140,"thread_start_us":97,"threads_started":1}
I20260621 05:10:46.037159 10975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:10:46.032189 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.131.130:34425 (local address 127.0.0.1:56808)
0621 05:10:46.032914 (+ 725us) negotiation.cc:107] Waiting for socket to connect
0621 05:10:46.032924 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0621 05:10:46.033004 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:10:46.033214 (+ 210us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:10:46.033214 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:10:46.033217 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 05:10:46.033250 (+ 33us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:10:46.033251 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:10:46.034479 (+ 1228us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:10:46.034480 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:10:46.035650 (+ 1170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:10:46.035652 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:10:46.036621 (+ 969us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:10:46.036622 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:10:46.036686 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:10:46.036689 (+ 3us) client_negotiation.cc:769] Sending connection context
0621 05:10:46.037032 (+ 343us) client_negotiation.cc:240] Negotiation successful
0621 05:10:46.037080 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":585,"mutex_wait_us":484,"thread_start_us":75,"threads_started":1}
I20260621 05:10:46.037324 10971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:10:46.032226 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.131.129:38671 (local address 127.0.0.1:56678)
0621 05:10:46.032366 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0621 05:10:46.032377 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0621 05:10:46.032631 (+ 254us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:10:46.032853 (+ 222us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:10:46.032853 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:10:46.032856 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 05:10:46.032904 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:10:46.032906 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:10:46.034509 (+ 1603us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:10:46.034510 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:10:46.035459 (+ 949us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:10:46.035462 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:10:46.036461 (+ 999us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:10:46.036462 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:10:46.036684 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:10:46.036691 (+ 7us) client_negotiation.cc:769] Sending connection context
0621 05:10:46.036740 (+ 49us) client_negotiation.cc:240] Negotiation successful
0621 05:10:46.036788 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":94,"threads_started":1}
I20260621 05:10:46.037209 10884 tablet_service.cc:1467] Processing CreateTablet for tablet b10a3caf182c473f901e4f62282755b9 (DEFAULT_TABLE table=test-table [id=912593bf2e4c4ff98872ec287b9ccc58]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 05:10:46.037619 10884 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b10a3caf182c473f901e4f62282755b9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 05:10:46.038046 10611 tablet_service.cc:1467] Processing CreateTablet for tablet b10a3caf182c473f901e4f62282755b9 (DEFAULT_TABLE table=test-table [id=912593bf2e4c4ff98872ec287b9ccc58]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 05:10:46.038317 10611 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b10a3caf182c473f901e4f62282755b9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 05:10:46.040678 10976 tablet_bootstrap.cc:492] T b10a3caf182c473f901e4f62282755b9 P 7e1eb017cbe947a08ad9b9e092651410: Bootstrap starting.
I20260621 05:10:46.041222 10977 tablet_bootstrap.cc:492] T b10a3caf182c473f901e4f62282755b9 P e3bf3241a75e43fca924eac143beb63f: Bootstrap starting.
I20260621 05:10:46.041468 10976 tablet_bootstrap.cc:654] T b10a3caf182c473f901e4f62282755b9 P 7e1eb017cbe947a08ad9b9e092651410: Neither blocks nor log segments found. Creating new log.
I20260621 05:10:46.041373 10748 tablet_service.cc:1467] Processing CreateTablet for tablet b10a3caf182c473f901e4f62282755b9 (DEFAULT_TABLE table=test-table [id=912593bf2e4c4ff98872ec287b9ccc58]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 05:10:46.041703 10748 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b10a3caf182c473f901e4f62282755b9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 05:10:46.041814 10976 log.cc:824] T b10a3caf182c473f901e4f62282755b9 P 7e1eb017cbe947a08ad9b9e092651410: Log is configured to *not* fsync() on all Append() calls
I20260621 05:10:46.041915 10977 tablet_bootstrap.cc:654] T b10a3caf182c473f901e4f62282755b9 P e3bf3241a75e43fca924eac143beb63f: Neither blocks nor log segments found. Creating new log.
I20260621 05:10:46.042284 10977 log.cc:824] T b10a3caf182c473f901e4f62282755b9 P e3bf3241a75e43fca924eac143beb63f: Log is configured to *not* fsync() on all Append() calls
I20260621 05:10:46.042649 10976 tablet_bootstrap.cc:492] T b10a3caf182c473f901e4f62282755b9 P 7e1eb017cbe947a08ad9b9e092651410: No bootstrap required, opened a new log
I20260621 05:10:46.042708 10976 ts_tablet_manager.cc:1397] T b10a3caf182c473f901e4f62282755b9 P 7e1eb017cbe947a08ad9b9e092651410: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260621 05:10:46.042958 10977 tablet_bootstrap.cc:492] T b10a3caf182c473f901e4f62282755b9 P e3bf3241a75e43fca924eac143beb63f: No bootstrap required, opened a new log
I20260621 05:10:46.043082 10977 ts_tablet_manager.cc:1397] T b10a3caf182c473f901e4f62282755b9 P e3bf3241a75e43fca924eac143beb63f: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260621 05:10:46.044699 10977 raft_consensus.cc:348] T b10a3caf182c473f901e4f62282755b9 P e3bf3241a75e43fca924eac143beb63f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8243bd6da6a04cdb80f9f0c681600639" member_type: VOTER last_known_addr { host: "127.3.131.130" port: 34425 } } peers { permanent_uuid: "7e1eb017cbe947a08ad9b9e092651410" member_type: VOTER last_known_addr { host: "127.3.131.131" port: 37567 } } peers { permanent_uuid: "e3bf3241a75e43fca924eac143beb63f" member_type: VOTER last_known_addr { host: "127.3.131.129" port: 38671 } }
I20260621 05:10:46.044822 10977 raft_consensus.cc:374] T b10a3caf182c473f901e4f62282755b9 P e3bf3241a75e43fca924eac143beb63f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 05:10:46.044843 10977 raft_consensus.cc:729] T b10a3caf182c473f901e4f62282755b9 P e3bf3241a75e43fca924eac143beb63f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e3bf3241a75e43fca924eac143beb63f, State: Initialized, Role: FOLLOWER
I20260621 05:10:46.044718 10976 raft_consensus.cc:348] T b10a3caf182c473f901e4f62282755b9 P 7e1eb017cbe947a08ad9b9e092651410 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8243bd6da6a04cdb80f9f0c681600639" member_type: VOTER last_known_addr { host: "127.3.131.130" port: 34425 } } peers { permanent_uuid: "7e1eb017cbe947a08ad9b9e092651410" member_type: VOTER last_known_addr { host: "127.3.131.131" port: 37567 } } peers { permanent_uuid: "e3bf3241a75e43fca924eac143beb63f" member_type: VOTER last_known_addr { host: "127.3.131.129" port: 38671 } }
I20260621 05:10:46.044879 10976 raft_consensus.cc:374] T b10a3caf182c473f901e4f62282755b9 P 7e1eb017cbe947a08ad9b9e092651410 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 05:10:46.044914 10976 raft_consensus.cc:729] T b10a3caf182c473f901e4f62282755b9 P 7e1eb017cbe947a08ad9b9e092651410 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7e1eb017cbe947a08ad9b9e092651410, State: Initialized, Role: FOLLOWER
I20260621 05:10:46.044960 10977 consensus_queue.cc:260] T b10a3caf182c473f901e4f62282755b9 P e3bf3241a75e43fca924eac143beb63f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8243bd6da6a04cdb80f9f0c681600639" member_type: VOTER last_known_addr { host: "127.3.131.130" port: 34425 } } peers { permanent_uuid: "7e1eb017cbe947a08ad9b9e092651410" member_type: VOTER last_known_addr { host: "127.3.131.131" port: 37567 } } peers { permanent_uuid: "e3bf3241a75e43fca924eac143beb63f" member_type: VOTER last_known_addr { host: "127.3.131.129" port: 38671 } }
I20260621 05:10:46.045019 10976 consensus_queue.cc:260] T b10a3caf182c473f901e4f62282755b9 P 7e1eb017cbe947a08ad9b9e092651410 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8243bd6da6a04cdb80f9f0c681600639" member_type: VOTER last_known_addr { host: "127.3.131.130" port: 34425 } } peers { permanent_uuid: "7e1eb017cbe947a08ad9b9e092651410" member_type: VOTER last_known_addr { host: "127.3.131.131" port: 37567 } } peers { permanent_uuid: "e3bf3241a75e43fca924eac143beb63f" member_type: VOTER last_known_addr { host: "127.3.131.129" port: 38671 } }
I20260621 05:10:46.045259 10976 ts_tablet_manager.cc:1428] T b10a3caf182c473f901e4f62282755b9 P 7e1eb017cbe947a08ad9b9e092651410: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260621 05:10:46.045543 10951 heartbeater.cc:503] Master 127.3.131.190:33159 was elected leader, sending a full tablet report...
I20260621 05:10:46.045974 10678 heartbeater.cc:503] Master 127.3.131.190:33159 was elected leader, sending a full tablet report...
I20260621 05:10:46.045220 10977 ts_tablet_manager.cc:1428] T b10a3caf182c473f901e4f62282755b9 P e3bf3241a75e43fca924eac143beb63f: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260621 05:10:46.047005 10981 tablet_bootstrap.cc:492] T b10a3caf182c473f901e4f62282755b9 P 8243bd6da6a04cdb80f9f0c681600639: Bootstrap starting.
I20260621 05:10:46.047765 10981 tablet_bootstrap.cc:654] T b10a3caf182c473f901e4f62282755b9 P 8243bd6da6a04cdb80f9f0c681600639: Neither blocks nor log segments found. Creating new log.
I20260621 05:10:46.048089 10981 log.cc:824] T b10a3caf182c473f901e4f62282755b9 P 8243bd6da6a04cdb80f9f0c681600639: Log is configured to *not* fsync() on all Append() calls
I20260621 05:10:46.048831 10981 tablet_bootstrap.cc:492] T b10a3caf182c473f901e4f62282755b9 P 8243bd6da6a04cdb80f9f0c681600639: No bootstrap required, opened a new log
I20260621 05:10:46.048882 10981 ts_tablet_manager.cc:1397] T b10a3caf182c473f901e4f62282755b9 P 8243bd6da6a04cdb80f9f0c681600639: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260621 05:10:46.050709 10981 raft_consensus.cc:348] T b10a3caf182c473f901e4f62282755b9 P 8243bd6da6a04cdb80f9f0c681600639 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8243bd6da6a04cdb80f9f0c681600639" member_type: VOTER last_known_addr { host: "127.3.131.130" port: 34425 } } peers { permanent_uuid: "7e1eb017cbe947a08ad9b9e092651410" member_type: VOTER last_known_addr { host: "127.3.131.131" port: 37567 } } peers { permanent_uuid: "e3bf3241a75e43fca924eac143beb63f" member_type: VOTER last_known_addr { host: "127.3.131.129" port: 38671 } }
I20260621 05:10:46.050832 10981 raft_consensus.cc:374] T b10a3caf182c473f901e4f62282755b9 P 8243bd6da6a04cdb80f9f0c681600639 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 05:10:46.050853 10981 raft_consensus.cc:729] T b10a3caf182c473f901e4f62282755b9 P 8243bd6da6a04cdb80f9f0c681600639 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8243bd6da6a04cdb80f9f0c681600639, State: Initialized, Role: FOLLOWER
I20260621 05:10:46.050971 10981 consensus_queue.cc:260] T b10a3caf182c473f901e4f62282755b9 P 8243bd6da6a04cdb80f9f0c681600639 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8243bd6da6a04cdb80f9f0c681600639" member_type: VOTER last_known_addr { host: "127.3.131.130" port: 34425 } } peers { permanent_uuid: "7e1eb017cbe947a08ad9b9e092651410" member_type: VOTER last_known_addr { host: "127.3.131.131" port: 37567 } } peers { permanent_uuid: "e3bf3241a75e43fca924eac143beb63f" member_type: VOTER last_known_addr { host: "127.3.131.129" port: 38671 } }
I20260621 05:10:46.051268 10981 ts_tablet_manager.cc:1428] T b10a3caf182c473f901e4f62282755b9 P 8243bd6da6a04cdb80f9f0c681600639: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260621 05:10:46.051286 10815 heartbeater.cc:503] Master 127.3.131.190:33159 was elected leader, sending a full tablet report...
I20260621 05:10:46.080355 10984 raft_consensus.cc:482] T b10a3caf182c473f901e4f62282755b9 P 8243bd6da6a04cdb80f9f0c681600639 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260621 05:10:46.080482 10984 raft_consensus.cc:504] T b10a3caf182c473f901e4f62282755b9 P 8243bd6da6a04cdb80f9f0c681600639 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8243bd6da6a04cdb80f9f0c681600639" member_type: VOTER last_known_addr { host: "127.3.131.130" port: 34425 } } peers { permanent_uuid: "7e1eb017cbe947a08ad9b9e092651410" member_type: VOTER last_known_addr { host: "127.3.131.131" port: 37567 } } peers { permanent_uuid: "e3bf3241a75e43fca924eac143beb63f" member_type: VOTER last_known_addr { host: "127.3.131.129" port: 38671 } }
I20260621 05:10:46.080874 10984 leader_election.cc:288] T b10a3caf182c473f901e4f62282755b9 P 8243bd6da6a04cdb80f9f0c681600639 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7e1eb017cbe947a08ad9b9e092651410 (127.3.131.131:37567), e3bf3241a75e43fca924eac143beb63f (127.3.131.129:38671)
I20260621 05:10:46.084406 10819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:10:46.081192 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.131.129:38671 (local address 127.3.131.130:43495)
0621 05:10:46.081285 (+ 93us) negotiation.cc:107] Waiting for socket to connect
0621 05:10:46.081299 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0621 05:10:46.081388 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:10:46.081537 (+ 149us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:10:46.081538 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:10:46.081540 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 05:10:46.081624 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:10:46.081627 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:10:46.082341 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:10:46.082343 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:10:46.083462 (+ 1119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:10:46.083465 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:10:46.084091 (+ 626us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:10:46.084092 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:10:46.084192 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:10:46.084196 (+ 4us) client_negotiation.cc:769] Sending connection context
0621 05:10:46.084249 (+ 53us) client_negotiation.cc:240] Negotiation successful
0621 05:10:46.084298 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":65,"mutex_wait_us":1}
I20260621 05:10:46.084523 10818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:10:46.081192 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.131.131:37567 (local address 127.3.131.130:57333)
0621 05:10:46.081411 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0621 05:10:46.081422 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0621 05:10:46.081476 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:10:46.081621 (+ 145us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:10:46.081622 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:10:46.081623 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 05:10:46.081712 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:10:46.081713 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:10:46.082488 (+ 775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:10:46.082489 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:10:46.083397 (+ 908us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:10:46.083399 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:10:46.084154 (+ 755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:10:46.084155 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:10:46.084214 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:10:46.084217 (+ 3us) client_negotiation.cc:769] Sending connection context
0621 05:10:46.084392 (+ 175us) client_negotiation.cc:240] Negotiation successful
0621 05:10:46.084435 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":205,"mutex_wait_us":168}
I20260621 05:10:46.084668 10972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:10:46.081327 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.131.130:57333 (local address 127.3.131.131:37567)
0621 05:10:46.081401 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0621 05:10:46.081404 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0621 05:10:46.081484 (+ 80us) server_negotiation.cc:407] Connection header received
0621 05:10:46.081533 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:10:46.081534 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:10:46.081543 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:10:46.081602 (+ 59us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 05:10:46.081812 (+ 210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:10:46.082368 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:10:46.083523 (+ 1155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:10:46.084061 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:10:46.084137 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:10:46.084147 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0621 05:10:46.084538 (+ 391us) server_negotiation.cc:299] Negotiation successful
0621 05:10:46.084601 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260621 05:10:46.084668 10974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:10:46.081357 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.131.130:43495 (local address 127.3.131.129:38671)
0621 05:10:46.081408 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0621 05:10:46.081410 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:10:46.081422 (+ 12us) server_negotiation.cc:407] Connection header received
0621 05:10:46.081453 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:10:46.081454 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:10:46.081465 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:10:46.081523 (+ 58us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 05:10:46.081717 (+ 194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:10:46.082220 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:10:46.083567 (+ 1347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:10:46.084015 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:10:46.084074 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:10:46.084081 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0621 05:10:46.084414 (+ 333us) server_negotiation.cc:299] Negotiation successful
0621 05:10:46.084461 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260621 05:10:46.085258 10631 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b10a3caf182c473f901e4f62282755b9" candidate_uuid: "8243bd6da6a04cdb80f9f0c681600639" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e3bf3241a75e43fca924eac143beb63f" is_pre_election: true
I20260621 05:10:46.085417 10631 raft_consensus.cc:2454] T b10a3caf182c473f901e4f62282755b9 P e3bf3241a75e43fca924eac143beb63f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8243bd6da6a04cdb80f9f0c681600639 in term 0.
I20260621 05:10:46.085377 10904 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b10a3caf182c473f901e4f62282755b9" candidate_uuid: "8243bd6da6a04cdb80f9f0c681600639" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7e1eb017cbe947a08ad9b9e092651410" is_pre_election: true
I20260621 05:10:46.085546 10904 raft_consensus.cc:2454] T b10a3caf182c473f901e4f62282755b9 P 7e1eb017cbe947a08ad9b9e092651410 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8243bd6da6a04cdb80f9f0c681600639 in term 0.
I20260621 05:10:46.085678 10702 leader_election.cc:302] T b10a3caf182c473f901e4f62282755b9 P 8243bd6da6a04cdb80f9f0c681600639 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8243bd6da6a04cdb80f9f0c681600639, e3bf3241a75e43fca924eac143beb63f; no voters:
I20260621 05:10:46.085829 10984 raft_consensus.cc:2781] T b10a3caf182c473f901e4f62282755b9 P 8243bd6da6a04cdb80f9f0c681600639 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260621 05:10:46.085886 10984 raft_consensus.cc:482] T b10a3caf182c473f901e4f62282755b9 P 8243bd6da6a04cdb80f9f0c681600639 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260621 05:10:46.085908 10984 raft_consensus.cc:3037] T b10a3caf182c473f901e4f62282755b9 P 8243bd6da6a04cdb80f9f0c681600639 [term 0 FOLLOWER]: Advancing to term 1
I20260621 05:10:46.086622 10984 raft_consensus.cc:504] T b10a3caf182c473f901e4f62282755b9 P 8243bd6da6a04cdb80f9f0c681600639 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8243bd6da6a04cdb80f9f0c681600639" member_type: VOTER last_known_addr { host: "127.3.131.130" port: 34425 } } peers { permanent_uuid: "7e1eb017cbe947a08ad9b9e092651410" member_type: VOTER last_known_addr { host: "127.3.131.131" port: 37567 } } peers { permanent_uuid: "e3bf3241a75e43fca924eac143beb63f" member_type: VOTER last_known_addr { host: "127.3.131.129" port: 38671 } }
I20260621 05:10:46.086766 10984 leader_election.cc:288] T b10a3caf182c473f901e4f62282755b9 P 8243bd6da6a04cdb80f9f0c681600639 [CANDIDATE]: Term 1 election: Requested vote from peers 7e1eb017cbe947a08ad9b9e092651410 (127.3.131.131:37567), e3bf3241a75e43fca924eac143beb63f (127.3.131.129:38671)
I20260621 05:10:46.086938 10904 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b10a3caf182c473f901e4f62282755b9" candidate_uuid: "8243bd6da6a04cdb80f9f0c681600639" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7e1eb017cbe947a08ad9b9e092651410"
I20260621 05:10:46.087015 10904 raft_consensus.cc:3037] T b10a3caf182c473f901e4f62282755b9 P 7e1eb017cbe947a08ad9b9e092651410 [term 0 FOLLOWER]: Advancing to term 1
I20260621 05:10:46.086941 10631 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b10a3caf182c473f901e4f62282755b9" candidate_uuid: "8243bd6da6a04cdb80f9f0c681600639" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e3bf3241a75e43fca924eac143beb63f"
I20260621 05:10:46.087092 10631 raft_consensus.cc:3037] T b10a3caf182c473f901e4f62282755b9 P e3bf3241a75e43fca924eac143beb63f [term 0 FOLLOWER]: Advancing to term 1
I20260621 05:10:46.087741 10904 raft_consensus.cc:2454] T b10a3caf182c473f901e4f62282755b9 P 7e1eb017cbe947a08ad9b9e092651410 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8243bd6da6a04cdb80f9f0c681600639 in term 1.
I20260621 05:10:46.087913 10701 leader_election.cc:302] T b10a3caf182c473f901e4f62282755b9 P 8243bd6da6a04cdb80f9f0c681600639 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7e1eb017cbe947a08ad9b9e092651410, 8243bd6da6a04cdb80f9f0c681600639; no voters:
I20260621 05:10:46.088030 10984 raft_consensus.cc:2781] T b10a3caf182c473f901e4f62282755b9 P 8243bd6da6a04cdb80f9f0c681600639 [term 1 FOLLOWER]: Leader election won for term 1
I20260621 05:10:46.088053 10631 raft_consensus.cc:2454] T b10a3caf182c473f901e4f62282755b9 P e3bf3241a75e43fca924eac143beb63f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8243bd6da6a04cdb80f9f0c681600639 in term 1.
I20260621 05:10:46.088241 10984 raft_consensus.cc:686] T b10a3caf182c473f901e4f62282755b9 P 8243bd6da6a04cdb80f9f0c681600639 [term 1 LEADER]: Becoming Leader. State: Replica: 8243bd6da6a04cdb80f9f0c681600639, State: Running, Role: LEADER
I20260621 05:10:46.088462 10984 consensus_queue.cc:237] T b10a3caf182c473f901e4f62282755b9 P 8243bd6da6a04cdb80f9f0c681600639 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8243bd6da6a04cdb80f9f0c681600639" member_type: VOTER last_known_addr { host: "127.3.131.130" port: 34425 } } peers { permanent_uuid: "7e1eb017cbe947a08ad9b9e092651410" member_type: VOTER last_known_addr { host: "127.3.131.131" port: 37567 } } peers { permanent_uuid: "e3bf3241a75e43fca924eac143beb63f" member_type: VOTER last_known_addr { host: "127.3.131.129" port: 38671 } }
I20260621 05:10:46.089321 9712 catalog_manager.cc:5495] T b10a3caf182c473f901e4f62282755b9 P 8243bd6da6a04cdb80f9f0c681600639 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8243bd6da6a04cdb80f9f0c681600639 (127.3.131.130). New cstate: current_term: 1 leader_uuid: "8243bd6da6a04cdb80f9f0c681600639" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8243bd6da6a04cdb80f9f0c681600639" member_type: VOTER last_known_addr { host: "127.3.131.130" port: 34425 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7e1eb017cbe947a08ad9b9e092651410" member_type: VOTER last_known_addr { host: "127.3.131.131" port: 37567 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e3bf3241a75e43fca924eac143beb63f" member_type: VOTER last_known_addr { host: "127.3.131.129" port: 38671 } health_report { overall_health: UNKNOWN } } }
I20260621 05:10:46.110298 10973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:10:46.107641 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56820 (local address 127.3.131.130:34425)
0621 05:10:46.107720 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0621 05:10:46.107722 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:10:46.107729 (+ 7us) server_negotiation.cc:407] Connection header received
0621 05:10:46.107764 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:10:46.107765 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:10:46.107782 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:10:46.107845 (+ 63us) server_negotiation.cc:226] Negotiated authn=TOKEN
0621 05:10:46.108018 (+ 173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:10:46.108534 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:10:46.109316 (+ 782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:10:46.109501 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:10:46.109557 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:10:46.109847 (+ 290us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0621 05:10:46.109999 (+ 152us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0621 05:10:46.110105 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0621 05:10:46.110175 (+ 70us) server_negotiation.cc:299] Negotiation successful
0621 05:10:46.110202 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260621 05:10:46.112319 10631 raft_consensus.cc:1261] T b10a3caf182c473f901e4f62282755b9 P e3bf3241a75e43fca924eac143beb63f [term 1 FOLLOWER]: Refusing update from remote peer 8243bd6da6a04cdb80f9f0c681600639: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 05:10:46.112366 10904 raft_consensus.cc:1261] T b10a3caf182c473f901e4f62282755b9 P 7e1eb017cbe947a08ad9b9e092651410 [term 1 FOLLOWER]: Refusing update from remote peer 8243bd6da6a04cdb80f9f0c681600639: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 05:10:46.112726 10985 consensus_queue.cc:1035] T b10a3caf182c473f901e4f62282755b9 P 8243bd6da6a04cdb80f9f0c681600639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3bf3241a75e43fca924eac143beb63f" member_type: VOTER last_known_addr { host: "127.3.131.129" port: 38671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 05:10:46.112855 10984 consensus_queue.cc:1035] T b10a3caf182c473f901e4f62282755b9 P 8243bd6da6a04cdb80f9f0c681600639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7e1eb017cbe947a08ad9b9e092651410" member_type: VOTER last_known_addr { host: "127.3.131.131" port: 37567 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 05:10:46.113051 10971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:10:46.109930 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.131.188:32887 (local address 127.0.0.1:35460)
0621 05:10:46.109996 (+ 66us) negotiation.cc:107] Waiting for socket to connect
0621 05:10:46.110031 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0621 05:10:46.110081 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:10:46.110293 (+ 212us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:10:46.110293 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:10:46.110296 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 05:10:46.110344 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:10:46.110346 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:10:46.111078 (+ 732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:10:46.111078 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:10:46.111972 (+ 894us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:10:46.111976 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:10:46.112736 (+ 760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:10:46.112737 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:10:46.112822 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:10:46.112825 (+ 3us) client_negotiation.cc:769] Sending connection context
0621 05:10:46.112918 (+ 93us) client_negotiation.cc:240] Negotiation successful
0621 05:10:46.112972 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":42}
I20260621 05:10:46.113081 10965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:10:46.110084 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35460 (local address 127.3.131.188:32887)
0621 05:10:46.110155 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0621 05:10:46.110158 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0621 05:10:46.110166 (+ 8us) server_negotiation.cc:407] Connection header received
0621 05:10:46.110195 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:10:46.110196 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:10:46.110219 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:10:46.110288 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 05:10:46.110451 (+ 163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:10:46.110985 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:10:46.112076 (+ 1091us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:10:46.112658 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:10:46.112733 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:10:46.112811 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0621 05:10:46.112924 (+ 113us) server_negotiation.cc:299] Negotiation successful
0621 05:10:46.112986 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260621 05:10:46.113452 10511 raft_consensus.cc:1203] T 00000000000000000000000000000000 P e6286f69e19545e883f56d392e7b8ea9 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.10] Dedup: 1.7->[1.8-1.10]
I20260621 05:10:46.114300 10545 sys_catalog.cc:455] T 00000000000000000000000000000000 P e6286f69e19545e883f56d392e7b8ea9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 08adad2fec7f4618bfbb0504365c5ecc. Latest consensus state: current_term: 1 leader_uuid: "08adad2fec7f4618bfbb0504365c5ecc" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "08adad2fec7f4618bfbb0504365c5ecc" member_type: VOTER last_known_addr { host: "127.3.131.190" port: 33159 } } peers { permanent_uuid: "65fa57b1227645fa8b95922a4da9e2ef" member_type: VOTER last_known_addr { host: "127.3.131.189" port: 41383 } attrs { promote: false } } peers { permanent_uuid: "e6286f69e19545e883f56d392e7b8ea9" member_type: NON_VOTER last_known_addr { host: "127.3.131.188" port: 32887 } attrs { promote: true } } }
I20260621 05:10:46.114396 10545 sys_catalog.cc:458] T 00000000000000000000000000000000 P e6286f69e19545e883f56d392e7b8ea9 [sys.catalog]: This master's current role is: LEARNER
W20260621 05:10:46.115715 10817 tablet.cc:2369] T b10a3caf182c473f901e4f62282755b9 P 8243bd6da6a04cdb80f9f0c681600639: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260621 05:10:46.116451 10989 mvcc.cc:205] Tried to move back new op lower bound from 7299148374473138176 to 7299148374379028480. Current Snapshot: MvccSnapshot[applied={T|T < 7299148374473138176}]
I20260621 05:10:46.117033 10991 mvcc.cc:205] Tried to move back new op lower bound from 7299148374473138176 to 7299148374379028480. Current Snapshot: MvccSnapshot[applied={T|T < 7299148374473138176}]
I20260621 05:10:46.118211 10988 mvcc.cc:205] Tried to move back new op lower bound from 7299148374473138176 to 7299148374379028480. Current Snapshot: MvccSnapshot[applied={T|T < 7299148374473138176 or (T in {7299148374473138176})}]
I20260621 05:10:46.120182 9712 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53638:
table { table_name: "test-table" } modify_external_catalogs: true
I20260621 05:10:46.120286 9712 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53638:
table { table_name: "test-table" } modify_external_catalogs: true
I20260621 05:10:46.120927 10272 sys_catalog.cc:455] T 00000000000000000000000000000000 P 08adad2fec7f4618bfbb0504365c5ecc [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "08adad2fec7f4618bfbb0504365c5ecc" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "08adad2fec7f4618bfbb0504365c5ecc" member_type: VOTER last_known_addr { host: "127.3.131.190" port: 33159 } } peers { permanent_uuid: "65fa57b1227645fa8b95922a4da9e2ef" member_type: VOTER last_known_addr { host: "127.3.131.189" port: 41383 } attrs { promote: false } } peers { permanent_uuid: "e6286f69e19545e883f56d392e7b8ea9" member_type: NON_VOTER last_known_addr { host: "127.3.131.188" port: 32887 } attrs { promote: true } } }
I20260621 05:10:46.121063 10272 sys_catalog.cc:458] T 00000000000000000000000000000000 P 08adad2fec7f4618bfbb0504365c5ecc [sys.catalog]: This master's current role is: LEADER
I20260621 05:10:46.121670 10272 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 08adad2fec7f4618bfbb0504365c5ecc: attempting to promote NON_VOTER e6286f69e19545e883f56d392e7b8ea9 to VOTER
I20260621 05:10:46.121784 10272 consensus_queue.cc:237] T 00000000000000000000000000000000 P 08adad2fec7f4618bfbb0504365c5ecc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 1.11, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 12 OBSOLETE_local: true peers { permanent_uuid: "08adad2fec7f4618bfbb0504365c5ecc" member_type: VOTER last_known_addr { host: "127.3.131.190" port: 33159 } } peers { permanent_uuid: "65fa57b1227645fa8b95922a4da9e2ef" member_type: VOTER last_known_addr { host: "127.3.131.189" port: 41383 } attrs { promote: false } } peers { permanent_uuid: "e6286f69e19545e883f56d392e7b8ea9" member_type: VOTER last_known_addr { host: "127.3.131.188" port: 32887 } attrs { promote: false } }
I20260621 05:10:46.122157 9712 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 08adad2fec7f4618bfbb0504365c5ecc: Sending DeleteTablet for 3 replicas of tablet b10a3caf182c473f901e4f62282755b9
I20260621 05:10:46.122313 10511 raft_consensus.cc:1261] T 00000000000000000000000000000000 P e6286f69e19545e883f56d392e7b8ea9 [term 1 LEARNER]: Refusing update from remote peer 08adad2fec7f4618bfbb0504365c5ecc: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20260621 05:10:46.122466 10748 tablet_service.cc:1514] Processing DeleteTablet for tablet b10a3caf182c473f901e4f62282755b9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 05:10:46 UTC) from {username='oryx', principal='oryx/127.3.131.190@KRBTEST.COM'} at 127.0.0.1:56808
I20260621 05:10:46.122726 11003 tablet_replica.cc:330] T b10a3caf182c473f901e4f62282755b9 P 8243bd6da6a04cdb80f9f0c681600639: stopping tablet replica
I20260621 05:10:46.122895 11003 raft_consensus.cc:2229] T b10a3caf182c473f901e4f62282755b9 P 8243bd6da6a04cdb80f9f0c681600639 [term 1 LEADER]: Raft consensus shutting down.
I20260621 05:10:46.122896 10884 tablet_service.cc:1514] Processing DeleteTablet for tablet b10a3caf182c473f901e4f62282755b9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 05:10:46 UTC) from {username='oryx', principal='oryx/127.3.131.190@KRBTEST.COM'} at 127.0.0.1:44338
I20260621 05:10:46.123104 11003 raft_consensus.cc:2258] T b10a3caf182c473f901e4f62282755b9 P 8243bd6da6a04cdb80f9f0c681600639 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 05:10:46.123083 10611 tablet_service.cc:1514] Processing DeleteTablet for tablet b10a3caf182c473f901e4f62282755b9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 05:10:46 UTC) from {username='oryx', principal='oryx/127.3.131.190@KRBTEST.COM'} at 127.0.0.1:56678
I20260621 05:10:46.123476 10321 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 65fa57b1227645fa8b95922a4da9e2ef [term 1 FOLLOWER]: Refusing update from remote peer 08adad2fec7f4618bfbb0504365c5ecc: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20260621 05:10:46.123719 10461 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 08adad2fec7f4618bfbb0504365c5ecc [LEADER]: Connected to new peer: Peer: permanent_uuid: "65fa57b1227645fa8b95922a4da9e2ef" member_type: VOTER last_known_addr { host: "127.3.131.189" port: 41383 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20260621 05:10:46.123807 10272 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 08adad2fec7f4618bfbb0504365c5ecc [LEADER]: Connected to new peer: Peer: permanent_uuid: "e6286f69e19545e883f56d392e7b8ea9" member_type: VOTER last_known_addr { host: "127.3.131.188" port: 32887 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20260621 05:10:46.124032 3598 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWZOx2_/build/release/bin/kudu with pid 9681
I20260621 05:10:46.124838 11003 ts_tablet_manager.cc:1905] T b10a3caf182c473f901e4f62282755b9 P 8243bd6da6a04cdb80f9f0c681600639: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 05:10:46.126947 11003 ts_tablet_manager.cc:1918] T b10a3caf182c473f901e4f62282755b9 P 8243bd6da6a04cdb80f9f0c681600639: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260621 05:10:46.127029 11003 log.cc:1196] T b10a3caf182c473f901e4f62282755b9 P 8243bd6da6a04cdb80f9f0c681600639: Deleting WAL directory at /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782018616648578-3598-0/minicluster-data/ts-1/wal/wals/b10a3caf182c473f901e4f62282755b9
I20260621 05:10:46.127349 11003 ts_tablet_manager.cc:1939] T b10a3caf182c473f901e4f62282755b9 P 8243bd6da6a04cdb80f9f0c681600639: Deleting consensus metadata
W20260621 05:10:46.130213 10482 connection.cc:441] server connection from 127.0.0.1:35460 torn down before Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35460 (request call id 4) could send its response
W20260621 05:10:46.130282 10701 connection.cc:535] server connection from 127.0.0.1:56808 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260621 05:10:46.131292 11005 tablet_replica.cc:330] T b10a3caf182c473f901e4f62282755b9 P e3bf3241a75e43fca924eac143beb63f: stopping tablet replica
I20260621 05:10:46.131438 11005 raft_consensus.cc:2229] T b10a3caf182c473f901e4f62282755b9 P e3bf3241a75e43fca924eac143beb63f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260621 05:10:46.131582 11005 raft_consensus.cc:2258] T b10a3caf182c473f901e4f62282755b9 P e3bf3241a75e43fca924eac143beb63f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 05:10:46.131767 11004 tablet_replica.cc:330] T b10a3caf182c473f901e4f62282755b9 P 7e1eb017cbe947a08ad9b9e092651410: stopping tablet replica
I20260621 05:10:46.131891 11004 raft_consensus.cc:2229] T b10a3caf182c473f901e4f62282755b9 P 7e1eb017cbe947a08ad9b9e092651410 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260621 05:10:46.132001 11004 raft_consensus.cc:2258] T b10a3caf182c473f901e4f62282755b9 P 7e1eb017cbe947a08ad9b9e092651410 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 05:10:46.132212 11005 ts_tablet_manager.cc:1905] T b10a3caf182c473f901e4f62282755b9 P e3bf3241a75e43fca924eac143beb63f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 05:10:46.133996 11005 ts_tablet_manager.cc:1918] T b10a3caf182c473f901e4f62282755b9 P e3bf3241a75e43fca924eac143beb63f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260621 05:10:46.134081 11005 log.cc:1196] T b10a3caf182c473f901e4f62282755b9 P e3bf3241a75e43fca924eac143beb63f: Deleting WAL directory at /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782018616648578-3598-0/minicluster-data/ts-0/wal/wals/b10a3caf182c473f901e4f62282755b9
I20260621 05:10:46.134371 11005 ts_tablet_manager.cc:1939] T b10a3caf182c473f901e4f62282755b9 P e3bf3241a75e43fca924eac143beb63f: Deleting consensus metadata
W20260621 05:10:46.134991 10564 connection.cc:441] server connection from 127.0.0.1:56678 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56678 (request call id 1) could send its response
I20260621 05:10:46.135505 11004 ts_tablet_manager.cc:1905] T b10a3caf182c473f901e4f62282755b9 P 7e1eb017cbe947a08ad9b9e092651410: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 05:10:46.137579 11004 ts_tablet_manager.cc:1918] T b10a3caf182c473f901e4f62282755b9 P 7e1eb017cbe947a08ad9b9e092651410: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260621 05:10:46.137668 11004 log.cc:1196] T b10a3caf182c473f901e4f62282755b9 P 7e1eb017cbe947a08ad9b9e092651410: Deleting WAL directory at /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782018616648578-3598-0/minicluster-data/ts-2/wal/wals/b10a3caf182c473f901e4f62282755b9
I20260621 05:10:46.137964 11004 ts_tablet_manager.cc:1939] T b10a3caf182c473f901e4f62282755b9 P 7e1eb017cbe947a08ad9b9e092651410: Deleting consensus metadata
W20260621 05:10:46.138418 10840 connection.cc:441] server connection from 127.0.0.1:44338 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44338 (request call id 1) could send its response
I20260621 05:10:46.140115 10965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:10:46.123693 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35474 (local address 127.3.131.188:32887)
0621 05:10:46.123735 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0621 05:10:46.123738 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0621 05:10:46.132311 (+ 8573us) server_negotiation.cc:407] Connection header received
0621 05:10:46.132428 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:10:46.132429 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:10:46.132442 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:10:46.132491 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:10:46.133304 (+ 813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:10:46.134075 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:10:46.135210 (+ 1135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:10:46.135432 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:10:46.135513 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:10:46.136904 (+ 1391us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:10:46.136905 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:10:46.136906 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:10:46.136914 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:10:46.138730 (+ 1816us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:10:46.139109 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:10:46.139110 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:10:46.139110 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:10:46.139207 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:10:46.139433 (+ 226us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:10:46.139434 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:10:46.139435 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:10:46.139587 (+ 152us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:10:46.139675 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0621 05:10:46.139906 (+ 231us) server_negotiation.cc:299] Negotiation successful
0621 05:10:46.140001 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260621 05:10:46.146449 10682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:10:46.132247 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35484 (local address 127.3.131.188:32887)
0621 05:10:46.132294 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0621 05:10:46.132296 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:10:46.138482 (+ 6186us) server_negotiation.cc:407] Connection header received
0621 05:10:46.138584 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:10:46.138586 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:10:46.138597 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:10:46.138627 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:10:46.139704 (+ 1077us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:10:46.140471 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:10:46.141561 (+ 1090us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:10:46.141854 (+ 293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:10:46.141928 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:10:46.143630 (+ 1702us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:10:46.143631 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:10:46.143631 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:10:46.143639 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:10:46.145229 (+ 1590us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:10:46.145630 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:10:46.145631 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:10:46.145631 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:10:46.145692 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:10:46.145908 (+ 216us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:10:46.145909 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:10:46.145909 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:10:46.146028 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:10:46.146131 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0621 05:10:46.146248 (+ 117us) server_negotiation.cc:299] Negotiation successful
0621 05:10:46.146327 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260621 05:10:46.147876 10964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:10:46.139659 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35240 (local address 127.3.131.189:41383)
0621 05:10:46.139706 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0621 05:10:46.139708 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:10:46.140003 (+ 295us) server_negotiation.cc:407] Connection header received
0621 05:10:46.140110 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:10:46.140111 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:10:46.140128 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:10:46.140158 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:10:46.141126 (+ 968us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:10:46.141695 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:10:46.142777 (+ 1082us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:10:46.143155 (+ 378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:10:46.143286 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:10:46.144715 (+ 1429us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:10:46.144716 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:10:46.144716 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:10:46.144724 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:10:46.146416 (+ 1692us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:10:46.147011 (+ 595us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:10:46.147012 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:10:46.147012 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:10:46.147072 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:10:46.147298 (+ 226us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:10:46.147298 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:10:46.147299 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:10:46.147440 (+ 141us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:10:46.147540 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0621 05:10:46.147662 (+ 122us) server_negotiation.cc:299] Negotiation successful
0621 05:10:46.147758 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260621 05:10:47.117468 11014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:10:47.117096 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.131.190:33159 (local address 127.3.131.130:54693)
0621 05:10:47.117287 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0621 05:10:47.117401 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.131.190:33159: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":168,"thread_start_us":94,"threads_started":1}
W20260621 05:10:47.117774 10815 heartbeater.cc:644] Failed to heartbeat to 127.3.131.190:33159 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.3.131.190:33159: connect: Connection refused (error 111)
I20260621 05:10:47.117901 11015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:10:47.117627 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.131.190:33159 (local address 127.3.131.129:48409)
0621 05:10:47.117765 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0621 05:10:47.117848 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.131.190:33159: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":50,"threads_started":1}
W20260621 05:10:47.118065 10678 heartbeater.cc:644] Failed to heartbeat to 127.3.131.190:33159 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.3.131.190:33159: connect: Connection refused (error 111)
I20260621 05:10:47.119148 11016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:10:47.118824 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.131.190:33159 (local address 127.3.131.131:40529)
0621 05:10:47.118962 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0621 05:10:47.119101 (+ 139us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.131.190:33159: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":52,"threads_started":1}
W20260621 05:10:47.119459 10951 heartbeater.cc:644] Failed to heartbeat to 127.3.131.190:33159 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.3.131.190:33159: connect: Connection refused (error 111)
I20260621 05:10:47.472582 11017 raft_consensus.cc:482] T 00000000000000000000000000000000 P e6286f69e19545e883f56d392e7b8ea9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 08adad2fec7f4618bfbb0504365c5ecc)
I20260621 05:10:47.472699 11017 raft_consensus.cc:504] T 00000000000000000000000000000000 P e6286f69e19545e883f56d392e7b8ea9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 12 OBSOLETE_local: true peers { permanent_uuid: "08adad2fec7f4618bfbb0504365c5ecc" member_type: VOTER last_known_addr { host: "127.3.131.190" port: 33159 } } peers { permanent_uuid: "65fa57b1227645fa8b95922a4da9e2ef" member_type: VOTER last_known_addr { host: "127.3.131.189" port: 41383 } attrs { promote: false } } peers { permanent_uuid: "e6286f69e19545e883f56d392e7b8ea9" member_type: VOTER last_known_addr { host: "127.3.131.188" port: 32887 } attrs { promote: false } }
I20260621 05:10:47.472982 11017 leader_election.cc:288] T 00000000000000000000000000000000 P e6286f69e19545e883f56d392e7b8ea9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 08adad2fec7f4618bfbb0504365c5ecc (127.3.131.190:33159), 65fa57b1227645fa8b95922a4da9e2ef (127.3.131.189:41383)
I20260621 05:10:47.473479 11018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:10:47.473223 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.131.190:33159 (local address 127.0.0.1:54108)
0621 05:10:47.473358 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0621 05:10:47.473424 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.131.190:33159: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":86,"threads_started":1}
W20260621 05:10:47.473728 10483 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.131.190:33159: connect: Connection refused (error 111)
I20260621 05:10:47.474311 11018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:10:47.474168 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.131.190:33159 (local address 127.0.0.1:54110)
0621 05:10:47.474246 (+ 78us) negotiation.cc:107] Waiting for socket to connect
0621 05:10:47.474276 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.131.190:33159: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
W20260621 05:10:47.474457 10483 leader_election.cc:334] T 00000000000000000000000000000000 P e6286f69e19545e883f56d392e7b8ea9 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 08adad2fec7f4618bfbb0504365c5ecc (127.3.131.190:33159): Network error: Client connection negotiation failed: client connection to 127.3.131.190:33159: connect: Connection refused (error 111)
I20260621 05:10:47.476876 11019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:10:47.473430 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.131.189:41383 (local address 127.0.0.1:35256)
0621 05:10:47.473525 (+ 95us) negotiation.cc:107] Waiting for socket to connect
0621 05:10:47.473535 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0621 05:10:47.473595 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:10:47.473922 (+ 327us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:10:47.473923 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:10:47.473927 (+ 4us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 05:10:47.474012 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:10:47.474015 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:10:47.474876 (+ 861us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:10:47.474877 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:10:47.475914 (+ 1037us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:10:47.475917 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:10:47.476648 (+ 731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:10:47.476649 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:10:47.476702 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:10:47.476706 (+ 4us) client_negotiation.cc:769] Sending connection context
0621 05:10:47.476767 (+ 61us) client_negotiation.cc:240] Negotiation successful
0621 05:10:47.476805 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":83,"thread_start_us":47,"threads_started":1}
I20260621 05:10:47.477023 11020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:10:47.473612 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35256 (local address 127.3.131.189:41383)
0621 05:10:47.473731 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0621 05:10:47.473734 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0621 05:10:47.473744 (+ 10us) server_negotiation.cc:407] Connection header received
0621 05:10:47.473777 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:10:47.473778 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:10:47.473790 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:10:47.473883 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 05:10:47.474139 (+ 256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:10:47.474778 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:10:47.476038 (+ 1260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:10:47.476540 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:10:47.476609 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:10:47.476619 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0621 05:10:47.476869 (+ 250us) server_negotiation.cc:299] Negotiation successful
0621 05:10:47.476924 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":98,"thread_start_us":57,"threads_started":1}
I20260621 05:10:47.477268 10321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e6286f69e19545e883f56d392e7b8ea9" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "65fa57b1227645fa8b95922a4da9e2ef" is_pre_election: true
I20260621 05:10:47.477562 10483 leader_election.cc:302] T 00000000000000000000000000000000 P e6286f69e19545e883f56d392e7b8ea9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e6286f69e19545e883f56d392e7b8ea9; no voters: 08adad2fec7f4618bfbb0504365c5ecc, 65fa57b1227645fa8b95922a4da9e2ef
I20260621 05:10:47.477728 11017 raft_consensus.cc:2726] T 00000000000000000000000000000000 P e6286f69e19545e883f56d392e7b8ea9 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260621 05:10:47.622977 11022 raft_consensus.cc:482] T 00000000000000000000000000000000 P 65fa57b1227645fa8b95922a4da9e2ef [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 08adad2fec7f4618bfbb0504365c5ecc)
I20260621 05:10:47.623129 11022 raft_consensus.cc:504] T 00000000000000000000000000000000 P 65fa57b1227645fa8b95922a4da9e2ef [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "08adad2fec7f4618bfbb0504365c5ecc" member_type: VOTER last_known_addr { host: "127.3.131.190" port: 33159 } } peers { permanent_uuid: "65fa57b1227645fa8b95922a4da9e2ef" member_type: VOTER last_known_addr { host: "127.3.131.189" port: 41383 } attrs { promote: false } } peers { permanent_uuid: "e6286f69e19545e883f56d392e7b8ea9" member_type: NON_VOTER last_known_addr { host: "127.3.131.188" port: 32887 } attrs { promote: true } }
I20260621 05:10:47.623447 11022 leader_election.cc:288] T 00000000000000000000000000000000 P 65fa57b1227645fa8b95922a4da9e2ef [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 08adad2fec7f4618bfbb0504365c5ecc (127.3.131.190:33159)
I20260621 05:10:47.623929 11023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:10:47.623706 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.131.190:33159 (local address 127.0.0.1:54156)
0621 05:10:47.623822 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0621 05:10:47.623891 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.131.190:33159: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98,"thread_start_us":73,"threads_started":1}
W20260621 05:10:47.624087 10296 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.131.190:33159: connect: Connection refused (error 111)
I20260621 05:10:47.624650 11023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:10:47.624539 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.131.190:33159 (local address 127.0.0.1:54158)
0621 05:10:47.624591 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0621 05:10:47.624622 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.131.190:33159: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260621 05:10:47.624773 10296 leader_election.cc:334] T 00000000000000000000000000000000 P 65fa57b1227645fa8b95922a4da9e2ef [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 08adad2fec7f4618bfbb0504365c5ecc (127.3.131.190:33159): Network error: Client connection negotiation failed: client connection to 127.3.131.190:33159: connect: Connection refused (error 111)
I20260621 05:10:47.624858 10296 leader_election.cc:302] T 00000000000000000000000000000000 P 65fa57b1227645fa8b95922a4da9e2ef [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 2 responses out of 2 voters: 1 yes votes; 1 no votes. yes voters: 65fa57b1227645fa8b95922a4da9e2ef; no voters: 08adad2fec7f4618bfbb0504365c5ecc
I20260621 05:10:47.624951 11022 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 65fa57b1227645fa8b95922a4da9e2ef [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260621 05:10:48.118786 11025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:10:48.118380 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.131.190:33159 (local address 127.3.131.130:55579)
0621 05:10:48.118635 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0621 05:10:48.118702 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.131.190:33159: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":229,"thread_start_us":91,"threads_started":1}
I20260621 05:10:48.119249 11026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:10:48.118583 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.131.190:33159 (local address 127.3.131.129:33265)
0621 05:10:48.118944 (+ 361us) negotiation.cc:107] Waiting for socket to connect
0621 05:10:48.118994 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.131.190:33159: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":275,"thread_start_us":87,"threads_started":1}
I20260621 05:10:48.120146 11027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:10:48.119915 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.131.190:33159 (local address 127.3.131.131:46933)
0621 05:10:48.120045 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0621 05:10:48.120085 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.131.190:33159: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":86,"threads_started":1}
I20260621 05:10:49.119673 11028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:10:49.119372 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.131.190:33159 (local address 127.3.131.130:51163)
0621 05:10:49.119528 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0621 05:10:49.119579 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.131.190:33159: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":59,"threads_started":1}
I20260621 05:10:49.119926 11029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:10:49.119711 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.131.190:33159 (local address 127.3.131.129:34759)
0621 05:10:49.119842 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0621 05:10:49.119884 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.131.190:33159: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":68,"threads_started":1}
I20260621 05:10:49.120807 11030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:10:49.120641 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.131.190:33159 (local address 127.3.131.131:34315)
0621 05:10:49.120731 (+ 90us) negotiation.cc:107] Waiting for socket to connect
0621 05:10:49.120764 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.131.190:33159: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72,"thread_start_us":51,"threads_started":1}
I20260621 05:10:49.591831 11031 raft_consensus.cc:482] T 00000000000000000000000000000000 P e6286f69e19545e883f56d392e7b8ea9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 08adad2fec7f4618bfbb0504365c5ecc)
I20260621 05:10:49.591935 11031 raft_consensus.cc:504] T 00000000000000000000000000000000 P e6286f69e19545e883f56d392e7b8ea9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 12 OBSOLETE_local: true peers { permanent_uuid: "08adad2fec7f4618bfbb0504365c5ecc" member_type: VOTER last_known_addr { host: "127.3.131.190" port: 33159 } } peers { permanent_uuid: "65fa57b1227645fa8b95922a4da9e2ef" member_type: VOTER last_known_addr { host: "127.3.131.189" port: 41383 } attrs { promote: false } } peers { permanent_uuid: "e6286f69e19545e883f56d392e7b8ea9" member_type: VOTER last_known_addr { host: "127.3.131.188" port: 32887 } attrs { promote: false } }
I20260621 05:10:49.592152 11031 leader_election.cc:288] T 00000000000000000000000000000000 P e6286f69e19545e883f56d392e7b8ea9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 08adad2fec7f4618bfbb0504365c5ecc (127.3.131.190:33159), 65fa57b1227645fa8b95922a4da9e2ef (127.3.131.189:41383)
I20260621 05:10:49.592573 11032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:10:49.592352 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.131.190:33159 (local address 127.0.0.1:54414)
0621 05:10:49.592481 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0621 05:10:49.592526 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.131.190:33159: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":84,"threads_started":1}
I20260621 05:10:49.592679 10321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e6286f69e19545e883f56d392e7b8ea9" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "65fa57b1227645fa8b95922a4da9e2ef" is_pre_election: true
I20260621 05:10:49.592793 10321 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 65fa57b1227645fa8b95922a4da9e2ef [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e6286f69e19545e883f56d392e7b8ea9 in term 1.
I20260621 05:10:49.593068 10483 leader_election.cc:302] T 00000000000000000000000000000000 P e6286f69e19545e883f56d392e7b8ea9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 65fa57b1227645fa8b95922a4da9e2ef, e6286f69e19545e883f56d392e7b8ea9; no voters:
I20260621 05:10:49.593129 11032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:10:49.593012 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.131.190:33159 (local address 127.0.0.1:54426)
0621 05:10:49.593063 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0621 05:10:49.593102 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.131.190:33159: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
I20260621 05:10:49.593189 11031 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e6286f69e19545e883f56d392e7b8ea9 [term 1 FOLLOWER]: Leader pre-election won for term 2
W20260621 05:10:49.593236 10483 leader_election.cc:334] T 00000000000000000000000000000000 P e6286f69e19545e883f56d392e7b8ea9 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 08adad2fec7f4618bfbb0504365c5ecc (127.3.131.190:33159): Network error: Client connection negotiation failed: client connection to 127.3.131.190:33159: connect: Connection refused (error 111)
I20260621 05:10:49.593250 11031 raft_consensus.cc:482] T 00000000000000000000000000000000 P e6286f69e19545e883f56d392e7b8ea9 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 08adad2fec7f4618bfbb0504365c5ecc)
I20260621 05:10:49.593299 11031 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e6286f69e19545e883f56d392e7b8ea9 [term 1 FOLLOWER]: Advancing to term 2
I20260621 05:10:49.594008 11031 raft_consensus.cc:504] T 00000000000000000000000000000000 P e6286f69e19545e883f56d392e7b8ea9 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 12 OBSOLETE_local: true peers { permanent_uuid: "08adad2fec7f4618bfbb0504365c5ecc" member_type: VOTER last_known_addr { host: "127.3.131.190" port: 33159 } } peers { permanent_uuid: "65fa57b1227645fa8b95922a4da9e2ef" member_type: VOTER last_known_addr { host: "127.3.131.189" port: 41383 } attrs { promote: false } } peers { permanent_uuid: "e6286f69e19545e883f56d392e7b8ea9" member_type: VOTER last_known_addr { host: "127.3.131.188" port: 32887 } attrs { promote: false } }
I20260621 05:10:49.594201 11031 leader_election.cc:288] T 00000000000000000000000000000000 P e6286f69e19545e883f56d392e7b8ea9 [CANDIDATE]: Term 2 election: Requested vote from peers 08adad2fec7f4618bfbb0504365c5ecc (127.3.131.190:33159), 65fa57b1227645fa8b95922a4da9e2ef (127.3.131.189:41383)
I20260621 05:10:49.594460 11032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:10:49.594288 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.131.190:33159 (local address 127.0.0.1:54438)
0621 05:10:49.594368 (+ 80us) negotiation.cc:107] Waiting for socket to connect
0621 05:10:49.594413 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.131.190:33159: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
I20260621 05:10:49.594504 10321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e6286f69e19545e883f56d392e7b8ea9" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "65fa57b1227645fa8b95922a4da9e2ef"
I20260621 05:10:49.594614 10321 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 65fa57b1227645fa8b95922a4da9e2ef [term 1 FOLLOWER]: Advancing to term 2
I20260621 05:10:49.594904 11032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:10:49.594745 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.131.190:33159 (local address 127.0.0.1:54450)
0621 05:10:49.594813 (+ 68us) negotiation.cc:107] Waiting for socket to connect
0621 05:10:49.594877 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.131.190:33159: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260621 05:10:49.595005 10483 leader_election.cc:334] T 00000000000000000000000000000000 P e6286f69e19545e883f56d392e7b8ea9 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 08adad2fec7f4618bfbb0504365c5ecc (127.3.131.190:33159): Network error: Client connection negotiation failed: client connection to 127.3.131.190:33159: connect: Connection refused (error 111)
I20260621 05:10:49.595459 10321 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 65fa57b1227645fa8b95922a4da9e2ef [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e6286f69e19545e883f56d392e7b8ea9 in term 2.
I20260621 05:10:49.595652 10483 leader_election.cc:302] T 00000000000000000000000000000000 P e6286f69e19545e883f56d392e7b8ea9 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 65fa57b1227645fa8b95922a4da9e2ef, e6286f69e19545e883f56d392e7b8ea9; no voters: 08adad2fec7f4618bfbb0504365c5ecc
I20260621 05:10:49.595757 11031 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e6286f69e19545e883f56d392e7b8ea9 [term 2 FOLLOWER]: Leader election won for term 2
I20260621 05:10:49.595867 11031 raft_consensus.cc:686] T 00000000000000000000000000000000 P e6286f69e19545e883f56d392e7b8ea9 [term 2 LEADER]: Becoming Leader. State: Replica: e6286f69e19545e883f56d392e7b8ea9, State: Running, Role: LEADER
I20260621 05:10:49.595952 11031 consensus_queue.cc:237] T 00000000000000000000000000000000 P e6286f69e19545e883f56d392e7b8ea9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 1.12, Last appended by leader: 12, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 12 OBSOLETE_local: true peers { permanent_uuid: "08adad2fec7f4618bfbb0504365c5ecc" member_type: VOTER last_known_addr { host: "127.3.131.190" port: 33159 } } peers { permanent_uuid: "65fa57b1227645fa8b95922a4da9e2ef" member_type: VOTER last_known_addr { host: "127.3.131.189" port: 41383 } attrs { promote: false } } peers { permanent_uuid: "e6286f69e19545e883f56d392e7b8ea9" member_type: VOTER last_known_addr { host: "127.3.131.188" port: 32887 } attrs { promote: false } }
I20260621 05:10:49.596335 11034 sys_catalog.cc:455] T 00000000000000000000000000000000 P e6286f69e19545e883f56d392e7b8ea9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e6286f69e19545e883f56d392e7b8ea9. Latest consensus state: current_term: 2 leader_uuid: "e6286f69e19545e883f56d392e7b8ea9" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "08adad2fec7f4618bfbb0504365c5ecc" member_type: VOTER last_known_addr { host: "127.3.131.190" port: 33159 } } peers { permanent_uuid: "65fa57b1227645fa8b95922a4da9e2ef" member_type: VOTER last_known_addr { host: "127.3.131.189" port: 41383 } attrs { promote: false } } peers { permanent_uuid: "e6286f69e19545e883f56d392e7b8ea9" member_type: NON_VOTER last_known_addr { host: "127.3.131.188" port: 32887 } attrs { promote: true } } } pending_config { opid_index: 12 OBSOLETE_local: true peers { permanent_uuid: "08adad2fec7f4618bfbb0504365c5ecc" member_type: VOTER last_known_addr { host: "127.3.131.190" port: 33159 } } peers { permanent_uuid: "65fa57b1227645fa8b95922a4da9e2ef" member_type: VOTER last_known_addr { host: "127.3.131.189" port: 41383 } attrs { promote: false } } peers { permanent_uuid: "e6286f69e19545e883f56d392e7b8ea9" member_type: VOTER last_known_addr { host: "127.3.131.188" port: 32887 } attrs { promote: false } } }
I20260621 05:10:49.596429 11034 sys_catalog.cc:458] T 00000000000000000000000000000000 P e6286f69e19545e883f56d392e7b8ea9 [sys.catalog]: This master's current role is: LEADER
I20260621 05:10:49.596597 11036 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260621 05:10:49.597055 11036 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260621 05:10:49.597229 11036 catalog_manager.cc:1259] Loaded cluster ID: 48235deea928424ca976ad57e26663f4
I20260621 05:10:49.597285 11036 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260621 05:10:49.597483 11036 catalog_manager.cc:1504] Loading token signing keys...
I20260621 05:10:49.597584 11036 catalog_manager.cc:5879] T 00000000000000000000000000000000 P e6286f69e19545e883f56d392e7b8ea9: Loaded TSK: 0
I20260621 05:10:49.597741 11036 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260621 05:10:49.654624 10501 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35484:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260621 05:10:49.656514 10501 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260621 05:10:49.657717 10321 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 65fa57b1227645fa8b95922a4da9e2ef [term 2 FOLLOWER]: Refusing update from remote peer e6286f69e19545e883f56d392e7b8ea9: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 2 index: 14. (index mismatch)
I20260621 05:10:49.657734 11032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:10:49.657618 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.131.190:33159 (local address 127.0.0.1:54470)
0621 05:10:49.657674 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0621 05:10:49.657706 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.131.190:33159: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
I20260621 05:10:49.658007 11031 consensus_queue.cc:1035] T 00000000000000000000000000000000 P e6286f69e19545e883f56d392e7b8ea9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "65fa57b1227645fa8b95922a4da9e2ef" member_type: VOTER last_known_addr { host: "127.3.131.189" port: 41383 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 11, Time since last communication: 0.000s
I20260621 05:10:49.658141 11032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:10:49.658065 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.131.190:33159 (local address 127.0.0.1:54472)
0621 05:10:49.658100 (+ 35us) negotiation.cc:107] Waiting for socket to connect
0621 05:10:49.658124 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.131.190:33159: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":17}
W20260621 05:10:49.658344 10483 consensus_peers.cc:487] T 00000000000000000000000000000000 P e6286f69e19545e883f56d392e7b8ea9 -> Peer 08adad2fec7f4618bfbb0504365c5ecc (127.3.131.190:33159): Couldn't send request to peer 08adad2fec7f4618bfbb0504365c5ecc. Status: Network error: Client connection negotiation failed: client connection to 127.3.131.190:33159: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260621 05:10:49.658915 11038 sys_catalog.cc:455] T 00000000000000000000000000000000 P 65fa57b1227645fa8b95922a4da9e2ef [sys.catalog]: SysCatalogTable state changed. Reason: New leader e6286f69e19545e883f56d392e7b8ea9. Latest consensus state: current_term: 2 leader_uuid: "e6286f69e19545e883f56d392e7b8ea9" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "08adad2fec7f4618bfbb0504365c5ecc" member_type: VOTER last_known_addr { host: "127.3.131.190" port: 33159 } } peers { permanent_uuid: "65fa57b1227645fa8b95922a4da9e2ef" member_type: VOTER last_known_addr { host: "127.3.131.189" port: 41383 } attrs { promote: false } } peers { permanent_uuid: "e6286f69e19545e883f56d392e7b8ea9" member_type: NON_VOTER last_known_addr { host: "127.3.131.188" port: 32887 } attrs { promote: true } } } pending_config { opid_index: 12 OBSOLETE_local: true peers { permanent_uuid: "08adad2fec7f4618bfbb0504365c5ecc" member_type: VOTER last_known_addr { host: "127.3.131.190" port: 33159 } } peers { permanent_uuid: "65fa57b1227645fa8b95922a4da9e2ef" member_type: VOTER last_known_addr { host: "127.3.131.189" port: 41383 } attrs { promote: false } } peers { permanent_uuid: "e6286f69e19545e883f56d392e7b8ea9" member_type: VOTER last_known_addr { host: "127.3.131.188" port: 32887 } attrs { promote: false } } }
I20260621 05:10:49.659013 11038 sys_catalog.cc:458] T 00000000000000000000000000000000 P 65fa57b1227645fa8b95922a4da9e2ef [sys.catalog]: This master's current role is: FOLLOWER
I20260621 05:10:49.659386 11034 raft_consensus.cc:2932] T 00000000000000000000000000000000 P e6286f69e19545e883f56d392e7b8ea9 [term 2 LEADER]: Committing config change with OpId 1.12: config changed from index 7 to 12, e6286f69e19545e883f56d392e7b8ea9 (127.3.131.188) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: true peers { permanent_uuid: "08adad2fec7f4618bfbb0504365c5ecc" member_type: VOTER last_known_addr { host: "127.3.131.190" port: 33159 } } peers { permanent_uuid: "65fa57b1227645fa8b95922a4da9e2ef" member_type: VOTER last_known_addr { host: "127.3.131.189" port: 41383 } attrs { promote: false } } peers { permanent_uuid: "e6286f69e19545e883f56d392e7b8ea9" member_type: VOTER last_known_addr { host: "127.3.131.188" port: 32887 } attrs { promote: false } } }
I20260621 05:10:49.659508 10321 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 65fa57b1227645fa8b95922a4da9e2ef [term 2 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 7 to 12, e6286f69e19545e883f56d392e7b8ea9 (127.3.131.188) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: true peers { permanent_uuid: "08adad2fec7f4618bfbb0504365c5ecc" member_type: VOTER last_known_addr { host: "127.3.131.190" port: 33159 } } peers { permanent_uuid: "65fa57b1227645fa8b95922a4da9e2ef" member_type: VOTER last_known_addr { host: "127.3.131.189" port: 41383 } attrs { promote: false } } peers { permanent_uuid: "e6286f69e19545e883f56d392e7b8ea9" member_type: VOTER last_known_addr { host: "127.3.131.188" port: 32887 } attrs { promote: false } } }
I20260621 05:10:49.660390 11037 mvcc.cc:205] Tried to move back new op lower bound from 7299148388995416064 to 7299148388745498624. Current Snapshot: MvccSnapshot[applied={T|T < 7299148388995416064}]
I20260621 05:10:49.660499 11031 sys_catalog.cc:455] T 00000000000000000000000000000000 P e6286f69e19545e883f56d392e7b8ea9 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "e6286f69e19545e883f56d392e7b8ea9" committed_config { opid_index: 12 OBSOLETE_local: true peers { permanent_uuid: "08adad2fec7f4618bfbb0504365c5ecc" member_type: VOTER last_known_addr { host: "127.3.131.190" port: 33159 } } peers { permanent_uuid: "65fa57b1227645fa8b95922a4da9e2ef" member_type: VOTER last_known_addr { host: "127.3.131.189" port: 41383 } attrs { promote: false } } peers { permanent_uuid: "e6286f69e19545e883f56d392e7b8ea9" member_type: VOTER last_known_addr { host: "127.3.131.188" port: 32887 } attrs { promote: false } } }
I20260621 05:10:49.660513 11034 sys_catalog.cc:455] T 00000000000000000000000000000000 P e6286f69e19545e883f56d392e7b8ea9 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "e6286f69e19545e883f56d392e7b8ea9" committed_config { opid_index: 12 OBSOLETE_local: true peers { permanent_uuid: "08adad2fec7f4618bfbb0504365c5ecc" member_type: VOTER last_known_addr { host: "127.3.131.190" port: 33159 } } peers { permanent_uuid: "65fa57b1227645fa8b95922a4da9e2ef" member_type: VOTER last_known_addr { host: "127.3.131.189" port: 41383 } attrs { promote: false } } peers { permanent_uuid: "e6286f69e19545e883f56d392e7b8ea9" member_type: VOTER last_known_addr { host: "127.3.131.188" port: 32887 } attrs { promote: false } } }
I20260621 05:10:49.660624 11031 sys_catalog.cc:458] T 00000000000000000000000000000000 P e6286f69e19545e883f56d392e7b8ea9 [sys.catalog]: This master's current role is: LEADER
I20260621 05:10:49.660633 11034 sys_catalog.cc:458] T 00000000000000000000000000000000 P e6286f69e19545e883f56d392e7b8ea9 [sys.catalog]: This master's current role is: LEADER
I20260621 05:10:49.660576 11038 sys_catalog.cc:455] T 00000000000000000000000000000000 P 65fa57b1227645fa8b95922a4da9e2ef [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "e6286f69e19545e883f56d392e7b8ea9" committed_config { opid_index: 12 OBSOLETE_local: true peers { permanent_uuid: "08adad2fec7f4618bfbb0504365c5ecc" member_type: VOTER last_known_addr { host: "127.3.131.190" port: 33159 } } peers { permanent_uuid: "65fa57b1227645fa8b95922a4da9e2ef" member_type: VOTER last_known_addr { host: "127.3.131.189" port: 41383 } attrs { promote: false } } peers { permanent_uuid: "e6286f69e19545e883f56d392e7b8ea9" member_type: VOTER last_known_addr { host: "127.3.131.188" port: 32887 } attrs { promote: false } } }
I20260621 05:10:49.660707 11038 sys_catalog.cc:458] T 00000000000000000000000000000000 P 65fa57b1227645fa8b95922a4da9e2ef [sys.catalog]: This master's current role is: FOLLOWER
I20260621 05:10:49.660750 11038 sys_catalog.cc:455] T 00000000000000000000000000000000 P 65fa57b1227645fa8b95922a4da9e2ef [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "e6286f69e19545e883f56d392e7b8ea9" committed_config { opid_index: 12 OBSOLETE_local: true peers { permanent_uuid: "08adad2fec7f4618bfbb0504365c5ecc" member_type: VOTER last_known_addr { host: "127.3.131.190" port: 33159 } } peers { permanent_uuid: "65fa57b1227645fa8b95922a4da9e2ef" member_type: VOTER last_known_addr { host: "127.3.131.189" port: 41383 } attrs { promote: false } } peers { permanent_uuid: "e6286f69e19545e883f56d392e7b8ea9" member_type: VOTER last_known_addr { host: "127.3.131.188" port: 32887 } attrs { promote: false } } }
I20260621 05:10:49.660801 11038 sys_catalog.cc:458] T 00000000000000000000000000000000 P 65fa57b1227645fa8b95922a4da9e2ef [sys.catalog]: This master's current role is: FOLLOWER
I20260621 05:10:49.660863 11039 mvcc.cc:205] Tried to move back new op lower bound from 7299148388995416064 to 7299148388745498624. Current Snapshot: MvccSnapshot[applied={T|T < 7299148388995416064}]
I20260621 05:10:49.666435 11032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:10:49.663010 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.131.130:34425 (local address 127.0.0.1:56828)
0621 05:10:49.663057 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0621 05:10:49.663068 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0621 05:10:49.663118 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:10:49.663661 (+ 543us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:10:49.663661 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:10:49.663663 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 05:10:49.663711 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:10:49.663733 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:10:49.664721 (+ 988us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:10:49.664723 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:10:49.665569 (+ 846us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:10:49.665571 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:10:49.666208 (+ 637us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:10:49.666209 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:10:49.666277 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:10:49.666282 (+ 5us) client_negotiation.cc:769] Sending connection context
0621 05:10:49.666338 (+ 56us) client_negotiation.cc:240] Negotiation successful
0621 05:10:49.666376 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":28,"mutex_wait_us":3}
I20260621 05:10:49.666489 11046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:10:49.663078 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56828 (local address 127.3.131.130:34425)
0621 05:10:49.663507 (+ 429us) server_negotiation.cc:206] Beginning negotiation
0621 05:10:49.663510 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0621 05:10:49.663520 (+ 10us) server_negotiation.cc:407] Connection header received
0621 05:10:49.663544 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:10:49.663545 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:10:49.663558 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:10:49.663648 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 05:10:49.663811 (+ 163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:10:49.664610 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:10:49.665660 (+ 1050us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:10:49.666103 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:10:49.666171 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:10:49.666184 (+ 13us) server_negotiation.cc:1035] Waiting for connection context
0621 05:10:49.666364 (+ 180us) server_negotiation.cc:299] Negotiation successful
0621 05:10:49.666405 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":405,"thread_start_us":67,"threads_started":1}
I20260621 05:10:49.666803 10748 tablet_service.cc:1467] Processing CreateTablet for tablet deb84733bd574776b28bc7fd84ad2af8 (DEFAULT_TABLE table=test-table [id=5200a8fce05d4b0ba90ed6b5d566aa38]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 05:10:49.666958 10748 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet deb84733bd574776b28bc7fd84ad2af8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 05:10:49.667207 11047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:10:49.663081 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.131.129:38671 (local address 127.0.0.1:56684)
0621 05:10:49.663335 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0621 05:10:49.663347 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0621 05:10:49.663388 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:10:49.663825 (+ 437us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:10:49.663825 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:10:49.663827 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 05:10:49.663873 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:10:49.663875 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:10:49.664573 (+ 698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:10:49.664574 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:10:49.665985 (+ 1411us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:10:49.665988 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:10:49.666898 (+ 910us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:10:49.666899 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:10:49.666959 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:10:49.666964 (+ 5us) client_negotiation.cc:769] Sending connection context
0621 05:10:49.667036 (+ 72us) client_negotiation.cc:240] Negotiation successful
0621 05:10:49.667078 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":180,"mutex_wait_us":54,"thread_start_us":77,"threads_started":1}
I20260621 05:10:49.667482 11048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:10:49.663276 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56684 (local address 127.3.131.129:38671)
0621 05:10:49.663381 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0621 05:10:49.663383 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:10:49.663392 (+ 9us) server_negotiation.cc:407] Connection header received
0621 05:10:49.663585 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:10:49.663586 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:10:49.663597 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:10:49.663659 (+ 62us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 05:10:49.663935 (+ 276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:10:49.664478 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:10:49.666085 (+ 1607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:10:49.666715 (+ 630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:10:49.666782 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:10:49.666794 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0621 05:10:49.667350 (+ 556us) server_negotiation.cc:299] Negotiation successful
0621 05:10:49.667398 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83,"thread_start_us":54,"threads_started":1}
I20260621 05:10:49.667681 10611 tablet_service.cc:1467] Processing CreateTablet for tablet deb84733bd574776b28bc7fd84ad2af8 (DEFAULT_TABLE table=test-table [id=5200a8fce05d4b0ba90ed6b5d566aa38]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 05:10:49.667809 10611 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet deb84733bd574776b28bc7fd84ad2af8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 05:10:49.667989 11045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:10:49.663024 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.131.131:37567 (local address 127.0.0.1:44340)
0621 05:10:49.663198 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0621 05:10:49.663209 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0621 05:10:49.663258 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:10:49.663753 (+ 495us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:10:49.663754 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:10:49.663756 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 05:10:49.663795 (+ 39us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:10:49.663797 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:10:49.664714 (+ 917us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:10:49.664714 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:10:49.665986 (+ 1272us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:10:49.665988 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:10:49.667700 (+ 1712us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:10:49.667701 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:10:49.667772 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:10:49.667778 (+ 6us) client_negotiation.cc:769] Sending connection context
0621 05:10:49.667842 (+ 64us) client_negotiation.cc:240] Negotiation successful
0621 05:10:49.667896 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":134,"mutex_wait_us":31,"thread_start_us":97,"threads_started":1}
I20260621 05:10:49.668236 11049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:10:49.663276 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44340 (local address 127.3.131.131:37567)
0621 05:10:49.663454 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0621 05:10:49.663456 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:10:49.663464 (+ 8us) server_negotiation.cc:407] Connection header received
0621 05:10:49.663491 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:10:49.663492 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:10:49.663502 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:10:49.663723 (+ 221us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 05:10:49.663872 (+ 149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:10:49.664640 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:10:49.666843 (+ 2203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:10:49.667498 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:10:49.667570 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:10:49.667583 (+ 13us) server_negotiation.cc:1035] Waiting for connection context
0621 05:10:49.668097 (+ 514us) server_negotiation.cc:299] Negotiation successful
0621 05:10:49.668167 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":153,"thread_start_us":57,"threads_started":1}
I20260621 05:10:49.668685 11050 tablet_bootstrap.cc:492] T deb84733bd574776b28bc7fd84ad2af8 P 8243bd6da6a04cdb80f9f0c681600639: Bootstrap starting.
I20260621 05:10:49.669112 11050 tablet_bootstrap.cc:654] T deb84733bd574776b28bc7fd84ad2af8 P 8243bd6da6a04cdb80f9f0c681600639: Neither blocks nor log segments found. Creating new log.
I20260621 05:10:49.669378 11051 tablet_bootstrap.cc:492] T deb84733bd574776b28bc7fd84ad2af8 P e3bf3241a75e43fca924eac143beb63f: Bootstrap starting.
I20260621 05:10:49.669571 10884 tablet_service.cc:1467] Processing CreateTablet for tablet deb84733bd574776b28bc7fd84ad2af8 (DEFAULT_TABLE table=test-table [id=5200a8fce05d4b0ba90ed6b5d566aa38]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 05:10:49.669695 10884 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet deb84733bd574776b28bc7fd84ad2af8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 05:10:49.669941 11051 tablet_bootstrap.cc:654] T deb84733bd574776b28bc7fd84ad2af8 P e3bf3241a75e43fca924eac143beb63f: Neither blocks nor log segments found. Creating new log.
I20260621 05:10:49.670495 11051 tablet_bootstrap.cc:492] T deb84733bd574776b28bc7fd84ad2af8 P e3bf3241a75e43fca924eac143beb63f: No bootstrap required, opened a new log
I20260621 05:10:49.670562 11051 ts_tablet_manager.cc:1397] T deb84733bd574776b28bc7fd84ad2af8 P e3bf3241a75e43fca924eac143beb63f: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260621 05:10:49.670723 11051 raft_consensus.cc:348] T deb84733bd574776b28bc7fd84ad2af8 P e3bf3241a75e43fca924eac143beb63f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8243bd6da6a04cdb80f9f0c681600639" member_type: VOTER last_known_addr { host: "127.3.131.130" port: 34425 } } peers { permanent_uuid: "7e1eb017cbe947a08ad9b9e092651410" member_type: VOTER last_known_addr { host: "127.3.131.131" port: 37567 } } peers { permanent_uuid: "e3bf3241a75e43fca924eac143beb63f" member_type: VOTER last_known_addr { host: "127.3.131.129" port: 38671 } }
I20260621 05:10:49.670805 11051 raft_consensus.cc:374] T deb84733bd574776b28bc7fd84ad2af8 P e3bf3241a75e43fca924eac143beb63f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 05:10:49.670836 11051 raft_consensus.cc:729] T deb84733bd574776b28bc7fd84ad2af8 P e3bf3241a75e43fca924eac143beb63f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e3bf3241a75e43fca924eac143beb63f, State: Initialized, Role: FOLLOWER
I20260621 05:10:49.670886 11051 consensus_queue.cc:260] T deb84733bd574776b28bc7fd84ad2af8 P e3bf3241a75e43fca924eac143beb63f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8243bd6da6a04cdb80f9f0c681600639" member_type: VOTER last_known_addr { host: "127.3.131.130" port: 34425 } } peers { permanent_uuid: "7e1eb017cbe947a08ad9b9e092651410" member_type: VOTER last_known_addr { host: "127.3.131.131" port: 37567 } } peers { permanent_uuid: "e3bf3241a75e43fca924eac143beb63f" member_type: VOTER last_known_addr { host: "127.3.131.129" port: 38671 } }
I20260621 05:10:49.671048 11051 ts_tablet_manager.cc:1428] T deb84733bd574776b28bc7fd84ad2af8 P e3bf3241a75e43fca924eac143beb63f: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260621 05:10:49.671041 11054 tablet_bootstrap.cc:492] T deb84733bd574776b28bc7fd84ad2af8 P 7e1eb017cbe947a08ad9b9e092651410: Bootstrap starting.
I20260621 05:10:49.671566 11054 tablet_bootstrap.cc:654] T deb84733bd574776b28bc7fd84ad2af8 P 7e1eb017cbe947a08ad9b9e092651410: Neither blocks nor log segments found. Creating new log.
I20260621 05:10:49.671700 11056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:10:49.671451 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.131.190:33159 (local address 127.3.131.129:58301)
0621 05:10:49.671614 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0621 05:10:49.671657 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.131.190:33159: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":91,"threads_started":1}
I20260621 05:10:49.671903 11050 tablet_bootstrap.cc:492] T deb84733bd574776b28bc7fd84ad2af8 P 8243bd6da6a04cdb80f9f0c681600639: No bootstrap required, opened a new log
I20260621 05:10:49.671965 11050 ts_tablet_manager.cc:1397] T deb84733bd574776b28bc7fd84ad2af8 P 8243bd6da6a04cdb80f9f0c681600639: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260621 05:10:49.672106 11050 raft_consensus.cc:348] T deb84733bd574776b28bc7fd84ad2af8 P 8243bd6da6a04cdb80f9f0c681600639 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8243bd6da6a04cdb80f9f0c681600639" member_type: VOTER last_known_addr { host: "127.3.131.130" port: 34425 } } peers { permanent_uuid: "7e1eb017cbe947a08ad9b9e092651410" member_type: VOTER last_known_addr { host: "127.3.131.131" port: 37567 } } peers { permanent_uuid: "e3bf3241a75e43fca924eac143beb63f" member_type: VOTER last_known_addr { host: "127.3.131.129" port: 38671 } }
I20260621 05:10:49.672206 11050 raft_consensus.cc:374] T deb84733bd574776b28bc7fd84ad2af8 P 8243bd6da6a04cdb80f9f0c681600639 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 05:10:49.672202 11054 tablet_bootstrap.cc:492] T deb84733bd574776b28bc7fd84ad2af8 P 7e1eb017cbe947a08ad9b9e092651410: No bootstrap required, opened a new log
I20260621 05:10:49.672222 11050 raft_consensus.cc:729] T deb84733bd574776b28bc7fd84ad2af8 P 8243bd6da6a04cdb80f9f0c681600639 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8243bd6da6a04cdb80f9f0c681600639, State: Initialized, Role: FOLLOWER
I20260621 05:10:49.672250 11054 ts_tablet_manager.cc:1397] T deb84733bd574776b28bc7fd84ad2af8 P 7e1eb017cbe947a08ad9b9e092651410: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260621 05:10:49.672300 11050 consensus_queue.cc:260] T deb84733bd574776b28bc7fd84ad2af8 P 8243bd6da6a04cdb80f9f0c681600639 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8243bd6da6a04cdb80f9f0c681600639" member_type: VOTER last_known_addr { host: "127.3.131.130" port: 34425 } } peers { permanent_uuid: "7e1eb017cbe947a08ad9b9e092651410" member_type: VOTER last_known_addr { host: "127.3.131.131" port: 37567 } } peers { permanent_uuid: "e3bf3241a75e43fca924eac143beb63f" member_type: VOTER last_known_addr { host: "127.3.131.129" port: 38671 } }
I20260621 05:10:49.672369 11054 raft_consensus.cc:348] T deb84733bd574776b28bc7fd84ad2af8 P 7e1eb017cbe947a08ad9b9e092651410 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8243bd6da6a04cdb80f9f0c681600639" member_type: VOTER last_known_addr { host: "127.3.131.130" port: 34425 } } peers { permanent_uuid: "7e1eb017cbe947a08ad9b9e092651410" member_type: VOTER last_known_addr { host: "127.3.131.131" port: 37567 } } peers { permanent_uuid: "e3bf3241a75e43fca924eac143beb63f" member_type: VOTER last_known_addr { host: "127.3.131.129" port: 38671 } }
I20260621 05:10:49.672451 11054 raft_consensus.cc:374] T deb84733bd574776b28bc7fd84ad2af8 P 7e1eb017cbe947a08ad9b9e092651410 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 05:10:49.672469 11054 raft_consensus.cc:729] T deb84733bd574776b28bc7fd84ad2af8 P 7e1eb017cbe947a08ad9b9e092651410 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7e1eb017cbe947a08ad9b9e092651410, State: Initialized, Role: FOLLOWER
I20260621 05:10:49.672477 11050 ts_tablet_manager.cc:1428] T deb84733bd574776b28bc7fd84ad2af8 P 8243bd6da6a04cdb80f9f0c681600639: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260621 05:10:49.672535 11054 consensus_queue.cc:260] T deb84733bd574776b28bc7fd84ad2af8 P 7e1eb017cbe947a08ad9b9e092651410 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8243bd6da6a04cdb80f9f0c681600639" member_type: VOTER last_known_addr { host: "127.3.131.130" port: 34425 } } peers { permanent_uuid: "7e1eb017cbe947a08ad9b9e092651410" member_type: VOTER last_known_addr { host: "127.3.131.131" port: 37567 } } peers { permanent_uuid: "e3bf3241a75e43fca924eac143beb63f" member_type: VOTER last_known_addr { host: "127.3.131.129" port: 38671 } }
I20260621 05:10:49.672602 10677 heartbeater.cc:503] Master 127.3.131.188:32887 was elected leader, sending a full tablet report...
I20260621 05:10:49.672705 11054 ts_tablet_manager.cc:1428] T deb84733bd574776b28bc7fd84ad2af8 P 7e1eb017cbe947a08ad9b9e092651410: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260621 05:10:49.672931 11056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:10:49.672728 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.131.190:33159 (local address 127.3.131.129:41061)
0621 05:10:49.672870 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0621 05:10:49.672901 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.131.190:33159: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":127}
I20260621 05:10:49.673285 11060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:10:49.673068 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.131.190:33159 (local address 127.3.131.131:60067)
0621 05:10:49.673181 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0621 05:10:49.673240 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.131.190:33159: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":93,"thread_start_us":78,"threads_started":1}
I20260621 05:10:49.673918 10950 heartbeater.cc:503] Master 127.3.131.188:32887 was elected leader, sending a full tablet report...
I20260621 05:10:49.674468 11060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:10:49.674171 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.131.190:33159 (local address 127.3.131.131:60791)
0621 05:10:49.674407 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0621 05:10:49.674437 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.131.190:33159: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":197}
I20260621 05:10:49.674702 11061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:10:49.673970 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.131.190:33159 (local address 127.3.131.130:38957)
0621 05:10:49.674630 (+ 660us) negotiation.cc:107] Waiting for socket to connect
0621 05:10:49.674668 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.131.190:33159: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":635,"thread_start_us":80,"threads_started":1}
W20260621 05:10:49.732447 10680 tablet.cc:2369] T deb84733bd574776b28bc7fd84ad2af8 P e3bf3241a75e43fca924eac143beb63f: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260621 05:10:49.757103 10953 tablet.cc:2369] T deb84733bd574776b28bc7fd84ad2af8 P 7e1eb017cbe947a08ad9b9e092651410: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260621 05:10:49.853143 11058 raft_consensus.cc:482] T deb84733bd574776b28bc7fd84ad2af8 P 8243bd6da6a04cdb80f9f0c681600639 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260621 05:10:49.853245 11058 raft_consensus.cc:504] T deb84733bd574776b28bc7fd84ad2af8 P 8243bd6da6a04cdb80f9f0c681600639 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8243bd6da6a04cdb80f9f0c681600639" member_type: VOTER last_known_addr { host: "127.3.131.130" port: 34425 } } peers { permanent_uuid: "7e1eb017cbe947a08ad9b9e092651410" member_type: VOTER last_known_addr { host: "127.3.131.131" port: 37567 } } peers { permanent_uuid: "e3bf3241a75e43fca924eac143beb63f" member_type: VOTER last_known_addr { host: "127.3.131.129" port: 38671 } }
I20260621 05:10:49.853433 11058 leader_election.cc:288] T deb84733bd574776b28bc7fd84ad2af8 P 8243bd6da6a04cdb80f9f0c681600639 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7e1eb017cbe947a08ad9b9e092651410 (127.3.131.131:37567), e3bf3241a75e43fca924eac143beb63f (127.3.131.129:38671)
I20260621 05:10:49.853745 10631 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "deb84733bd574776b28bc7fd84ad2af8" candidate_uuid: "8243bd6da6a04cdb80f9f0c681600639" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e3bf3241a75e43fca924eac143beb63f" is_pre_election: true
I20260621 05:10:49.853828 10631 raft_consensus.cc:2454] T deb84733bd574776b28bc7fd84ad2af8 P e3bf3241a75e43fca924eac143beb63f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8243bd6da6a04cdb80f9f0c681600639 in term 0.
I20260621 05:10:49.853775 10904 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "deb84733bd574776b28bc7fd84ad2af8" candidate_uuid: "8243bd6da6a04cdb80f9f0c681600639" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7e1eb017cbe947a08ad9b9e092651410" is_pre_election: true
I20260621 05:10:49.853926 10904 raft_consensus.cc:2454] T deb84733bd574776b28bc7fd84ad2af8 P 7e1eb017cbe947a08ad9b9e092651410 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8243bd6da6a04cdb80f9f0c681600639 in term 0.
I20260621 05:10:49.854014 10702 leader_election.cc:302] T deb84733bd574776b28bc7fd84ad2af8 P 8243bd6da6a04cdb80f9f0c681600639 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8243bd6da6a04cdb80f9f0c681600639, e3bf3241a75e43fca924eac143beb63f; no voters:
I20260621 05:10:49.854159 11058 raft_consensus.cc:2781] T deb84733bd574776b28bc7fd84ad2af8 P 8243bd6da6a04cdb80f9f0c681600639 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260621 05:10:49.854216 11058 raft_consensus.cc:482] T deb84733bd574776b28bc7fd84ad2af8 P 8243bd6da6a04cdb80f9f0c681600639 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260621 05:10:49.854234 11058 raft_consensus.cc:3037] T deb84733bd574776b28bc7fd84ad2af8 P 8243bd6da6a04cdb80f9f0c681600639 [term 0 FOLLOWER]: Advancing to term 1
I20260621 05:10:49.854995 11058 raft_consensus.cc:504] T deb84733bd574776b28bc7fd84ad2af8 P 8243bd6da6a04cdb80f9f0c681600639 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8243bd6da6a04cdb80f9f0c681600639" member_type: VOTER last_known_addr { host: "127.3.131.130" port: 34425 } } peers { permanent_uuid: "7e1eb017cbe947a08ad9b9e092651410" member_type: VOTER last_known_addr { host: "127.3.131.131" port: 37567 } } peers { permanent_uuid: "e3bf3241a75e43fca924eac143beb63f" member_type: VOTER last_known_addr { host: "127.3.131.129" port: 38671 } }
I20260621 05:10:49.855162 11058 leader_election.cc:288] T deb84733bd574776b28bc7fd84ad2af8 P 8243bd6da6a04cdb80f9f0c681600639 [CANDIDATE]: Term 1 election: Requested vote from peers 7e1eb017cbe947a08ad9b9e092651410 (127.3.131.131:37567), e3bf3241a75e43fca924eac143beb63f (127.3.131.129:38671)
I20260621 05:10:49.855343 10904 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "deb84733bd574776b28bc7fd84ad2af8" candidate_uuid: "8243bd6da6a04cdb80f9f0c681600639" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7e1eb017cbe947a08ad9b9e092651410"
I20260621 05:10:49.855436 10904 raft_consensus.cc:3037] T deb84733bd574776b28bc7fd84ad2af8 P 7e1eb017cbe947a08ad9b9e092651410 [term 0 FOLLOWER]: Advancing to term 1
I20260621 05:10:49.855389 10631 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "deb84733bd574776b28bc7fd84ad2af8" candidate_uuid: "8243bd6da6a04cdb80f9f0c681600639" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e3bf3241a75e43fca924eac143beb63f"
I20260621 05:10:49.855506 10631 raft_consensus.cc:3037] T deb84733bd574776b28bc7fd84ad2af8 P e3bf3241a75e43fca924eac143beb63f [term 0 FOLLOWER]: Advancing to term 1
I20260621 05:10:49.856191 10904 raft_consensus.cc:2454] T deb84733bd574776b28bc7fd84ad2af8 P 7e1eb017cbe947a08ad9b9e092651410 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8243bd6da6a04cdb80f9f0c681600639 in term 1.
I20260621 05:10:49.856302 10631 raft_consensus.cc:2454] T deb84733bd574776b28bc7fd84ad2af8 P e3bf3241a75e43fca924eac143beb63f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8243bd6da6a04cdb80f9f0c681600639 in term 1.
I20260621 05:10:49.856418 10701 leader_election.cc:302] T deb84733bd574776b28bc7fd84ad2af8 P 8243bd6da6a04cdb80f9f0c681600639 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7e1eb017cbe947a08ad9b9e092651410, 8243bd6da6a04cdb80f9f0c681600639; no voters:
I20260621 05:10:49.856560 11058 raft_consensus.cc:2781] T deb84733bd574776b28bc7fd84ad2af8 P 8243bd6da6a04cdb80f9f0c681600639 [term 1 FOLLOWER]: Leader election won for term 1
I20260621 05:10:49.856774 11058 raft_consensus.cc:686] T deb84733bd574776b28bc7fd84ad2af8 P 8243bd6da6a04cdb80f9f0c681600639 [term 1 LEADER]: Becoming Leader. State: Replica: 8243bd6da6a04cdb80f9f0c681600639, State: Running, Role: LEADER
I20260621 05:10:49.856828 11058 consensus_queue.cc:237] T deb84733bd574776b28bc7fd84ad2af8 P 8243bd6da6a04cdb80f9f0c681600639 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8243bd6da6a04cdb80f9f0c681600639" member_type: VOTER last_known_addr { host: "127.3.131.130" port: 34425 } } peers { permanent_uuid: "7e1eb017cbe947a08ad9b9e092651410" member_type: VOTER last_known_addr { host: "127.3.131.131" port: 37567 } } peers { permanent_uuid: "e3bf3241a75e43fca924eac143beb63f" member_type: VOTER last_known_addr { host: "127.3.131.129" port: 38671 } }
I20260621 05:10:49.856887 10814 heartbeater.cc:503] Master 127.3.131.188:32887 was elected leader, sending a full tablet report...
I20260621 05:10:49.857167 11061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:10:49.857047 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.131.190:33159 (local address 127.3.131.130:44107)
0621 05:10:49.857089 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0621 05:10:49.857132 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.131.190:33159: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":19}
I20260621 05:10:49.857595 10493 catalog_manager.cc:5495] T deb84733bd574776b28bc7fd84ad2af8 P 8243bd6da6a04cdb80f9f0c681600639 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8243bd6da6a04cdb80f9f0c681600639 (127.3.131.130). New cstate: current_term: 1 leader_uuid: "8243bd6da6a04cdb80f9f0c681600639" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8243bd6da6a04cdb80f9f0c681600639" member_type: VOTER last_known_addr { host: "127.3.131.130" port: 34425 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7e1eb017cbe947a08ad9b9e092651410" member_type: VOTER last_known_addr { host: "127.3.131.131" port: 37567 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e3bf3241a75e43fca924eac143beb63f" member_type: VOTER last_known_addr { host: "127.3.131.129" port: 38671 } health_report { overall_health: UNKNOWN } } }
I20260621 05:10:49.898546 11046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:10:49.896106 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56834 (local address 127.3.131.130:34425)
0621 05:10:49.896181 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0621 05:10:49.896184 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0621 05:10:49.896194 (+ 10us) server_negotiation.cc:407] Connection header received
0621 05:10:49.896275 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:10:49.896276 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:10:49.896286 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:10:49.896341 (+ 55us) server_negotiation.cc:226] Negotiated authn=TOKEN
0621 05:10:49.896478 (+ 137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:10:49.896990 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:10:49.897826 (+ 836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:10:49.898023 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:10:49.898099 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:10:49.898269 (+ 170us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0621 05:10:49.898307 (+ 38us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0621 05:10:49.898367 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0621 05:10:49.898429 (+ 62us) server_negotiation.cc:299] Negotiation successful
0621 05:10:49.898452 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260621 05:10:49.899468 10631 raft_consensus.cc:1261] T deb84733bd574776b28bc7fd84ad2af8 P e3bf3241a75e43fca924eac143beb63f [term 1 FOLLOWER]: Refusing update from remote peer 8243bd6da6a04cdb80f9f0c681600639: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 05:10:49.899506 10904 raft_consensus.cc:1261] T deb84733bd574776b28bc7fd84ad2af8 P 7e1eb017cbe947a08ad9b9e092651410 [term 1 FOLLOWER]: Refusing update from remote peer 8243bd6da6a04cdb80f9f0c681600639: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 05:10:49.899809 11062 consensus_queue.cc:1035] T deb84733bd574776b28bc7fd84ad2af8 P 8243bd6da6a04cdb80f9f0c681600639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3bf3241a75e43fca924eac143beb63f" member_type: VOTER last_known_addr { host: "127.3.131.129" port: 38671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 05:10:49.899925 11058 consensus_queue.cc:1035] T deb84733bd574776b28bc7fd84ad2af8 P 8243bd6da6a04cdb80f9f0c681600639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7e1eb017cbe947a08ad9b9e092651410" member_type: VOTER last_known_addr { host: "127.3.131.131" port: 37567 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 05:10:49.900591 11060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:10:49.900472 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.131.190:33159 (local address 127.3.131.131:35055)
0621 05:10:49.900524 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0621 05:10:49.900556 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.131.190:33159: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":19}
I20260621 05:10:49.901327 11056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:10:49.901196 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.131.190:33159 (local address 127.3.131.129:52597)
0621 05:10:49.901264 (+ 68us) negotiation.cc:107] Waiting for socket to connect
0621 05:10:49.901293 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.131.190:33159: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260621 05:10:49.902302 11061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:10:49.902172 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.131.190:33159 (local address 127.3.131.130:41785)
0621 05:10:49.902235 (+ 63us) negotiation.cc:107] Waiting for socket to connect
0621 05:10:49.902269 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.131.190:33159: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
I20260621 05:10:49.902704 11061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:10:49.902519 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.131.190:33159 (local address 127.3.131.130:44137)
0621 05:10:49.902644 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0621 05:10:49.902674 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.131.190:33159: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107}
I20260621 05:10:49.902832 11060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:10:49.902711 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.131.190:33159 (local address 127.3.131.131:39297)
0621 05:10:49.902778 (+ 67us) negotiation.cc:107] Waiting for socket to connect
0621 05:10:49.902803 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.131.190:33159: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260621 05:10:49.903090 11056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:10:49.902912 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.131.190:33159 (local address 127.3.131.129:40237)
0621 05:10:49.903032 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0621 05:10:49.903060 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.131.190:33159: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":100}
I20260621 05:10:49.903891 10495 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35484:
table { table_name: "test-table" } modify_external_catalogs: true
I20260621 05:10:49.903996 10495 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35484:
table { table_name: "test-table" } modify_external_catalogs: true
I20260621 05:10:49.905601 10495 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e6286f69e19545e883f56d392e7b8ea9: Sending DeleteTablet for 3 replicas of tablet deb84733bd574776b28bc7fd84ad2af8
I20260621 05:10:49.905916 10748 tablet_service.cc:1514] Processing DeleteTablet for tablet deb84733bd574776b28bc7fd84ad2af8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 05:10:49 UTC) from {username='oryx', principal='oryx/127.3.131.188@KRBTEST.COM'} at 127.0.0.1:56828
I20260621 05:10:49.905931 10611 tablet_service.cc:1514] Processing DeleteTablet for tablet deb84733bd574776b28bc7fd84ad2af8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 05:10:49 UTC) from {username='oryx', principal='oryx/127.3.131.188@KRBTEST.COM'} at 127.0.0.1:56684
I20260621 05:10:49.905920 10884 tablet_service.cc:1514] Processing DeleteTablet for tablet deb84733bd574776b28bc7fd84ad2af8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 05:10:49 UTC) from {username='oryx', principal='oryx/127.3.131.188@KRBTEST.COM'} at 127.0.0.1:44340
I20260621 05:10:49.906198 11075 tablet_replica.cc:330] T deb84733bd574776b28bc7fd84ad2af8 P 8243bd6da6a04cdb80f9f0c681600639: stopping tablet replica
I20260621 05:10:49.906260 11075 raft_consensus.cc:2229] T deb84733bd574776b28bc7fd84ad2af8 P 8243bd6da6a04cdb80f9f0c681600639 [term 1 LEADER]: Raft consensus shutting down.
I20260621 05:10:49.906322 11075 raft_consensus.cc:2258] T deb84733bd574776b28bc7fd84ad2af8 P 8243bd6da6a04cdb80f9f0c681600639 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 05:10:49.906412 3598 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWZOx2_/build/release/bin/kudu with pid 10458
I20260621 05:10:49.906430 11073 tablet_replica.cc:330] T deb84733bd574776b28bc7fd84ad2af8 P 7e1eb017cbe947a08ad9b9e092651410: stopping tablet replica
I20260621 05:10:49.906481 11073 raft_consensus.cc:2229] T deb84733bd574776b28bc7fd84ad2af8 P 7e1eb017cbe947a08ad9b9e092651410 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260621 05:10:49.906549 11073 raft_consensus.cc:2258] T deb84733bd574776b28bc7fd84ad2af8 P 7e1eb017cbe947a08ad9b9e092651410 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 05:10:49.907294 11075 ts_tablet_manager.cc:1905] T deb84733bd574776b28bc7fd84ad2af8 P 8243bd6da6a04cdb80f9f0c681600639: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 05:10:49.907335 11073 ts_tablet_manager.cc:1905] T deb84733bd574776b28bc7fd84ad2af8 P 7e1eb017cbe947a08ad9b9e092651410: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 05:10:49.908783 11075 ts_tablet_manager.cc:1918] T deb84733bd574776b28bc7fd84ad2af8 P 8243bd6da6a04cdb80f9f0c681600639: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260621 05:10:49.908875 11075 log.cc:1196] T deb84733bd574776b28bc7fd84ad2af8 P 8243bd6da6a04cdb80f9f0c681600639: Deleting WAL directory at /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782018616648578-3598-0/minicluster-data/ts-1/wal/wals/deb84733bd574776b28bc7fd84ad2af8
I20260621 05:10:49.909168 11075 ts_tablet_manager.cc:1939] T deb84733bd574776b28bc7fd84ad2af8 P 8243bd6da6a04cdb80f9f0c681600639: Deleting consensus metadata
I20260621 05:10:49.909206 11073 ts_tablet_manager.cc:1918] T deb84733bd574776b28bc7fd84ad2af8 P 7e1eb017cbe947a08ad9b9e092651410: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260621 05:10:49.909258 11073 log.cc:1196] T deb84733bd574776b28bc7fd84ad2af8 P 7e1eb017cbe947a08ad9b9e092651410: Deleting WAL directory at /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782018616648578-3598-0/minicluster-data/ts-2/wal/wals/deb84733bd574776b28bc7fd84ad2af8
I20260621 05:10:49.909477 11073 ts_tablet_manager.cc:1939] T deb84733bd574776b28bc7fd84ad2af8 P 7e1eb017cbe947a08ad9b9e092651410: Deleting consensus metadata
I20260621 05:10:49.909513 10485 catalog_manager.cc:4841] TS 8243bd6da6a04cdb80f9f0c681600639 (127.3.131.130:34425): tablet deb84733bd574776b28bc7fd84ad2af8 (table test-table [id=5200a8fce05d4b0ba90ed6b5d566aa38]) successfully deleted
I20260621 05:10:49.910005 10482 catalog_manager.cc:4841] TS 7e1eb017cbe947a08ad9b9e092651410 (127.3.131.131:37567): tablet deb84733bd574776b28bc7fd84ad2af8 (table test-table [id=5200a8fce05d4b0ba90ed6b5d566aa38]) successfully deleted
I20260621 05:10:49.913844 11045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:10:49.913695 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.131.129:38671 (local address 127.0.0.1:56700)
0621 05:10:49.913764 (+ 69us) negotiation.cc:107] Waiting for socket to connect
0621 05:10:49.913799 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.131.129:38671: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
I20260621 05:10:49.913883 3598 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWZOx2_/build/release/bin/kudu with pid 10685
W20260621 05:10:49.914033 10483 catalog_manager.cc:4584] TS e3bf3241a75e43fca924eac143beb63f (127.3.131.129:38671): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet deb84733bd574776b28bc7fd84ad2af8: Network error: Client connection negotiation failed: client connection to 127.3.131.129:38671: connect: Connection refused (error 111)
I20260621 05:10:49.920506 3598 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWZOx2_/build/release/bin/kudu with pid 10821
I20260621 05:10:49.925760 3598 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWZOx2_/build/release/bin/kudu with pid 10177
I20260621 05:10:49.930864 3598 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWZOx2_/build/release/bin/kudu with pid 10365
2026-06-21T05:10:49Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (6843 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782018616648578-3598-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 21 05:10:49 dist-test-slave-vrq1 krb5kdc[11078](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 21 05:10:49 dist-test-slave-vrq1 krb5kdc[11078](info): set up 2 sockets
Jun 21 05:10:49 dist-test-slave-vrq1 krb5kdc[11078](info): commencing operation
krb5kdc: starting...
W20260621 05:10:50.988242 3598 mini_kdc.cc:121] Time spent starting KDC: real 1.034s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782018616648578-3598-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782018616648578-3598-0/krb5kdc/test-user.keytab.
Jun 21 05:10:51 dist-test-slave-vrq1 krb5kdc[11078](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782018651, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-21T05:10:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-21T05:10:51Z Disabled control of system clock
WARNING: no policy specified for oryx/127.3.131.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.3.131.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.3.131.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782018616648578-3598-0/krb5kdc/oryx_127.3.131.190.keytab.
Entry for principal oryx/127.3.131.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782018616648578-3598-0/krb5kdc/oryx_127.3.131.190.keytab.
I20260621 05:10:51.119436 3598 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWZOx2_/build/release/bin/kudu
/tmp/dist-test-taskWZOx2_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782018616648578-3598-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782018616648578-3598-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782018616648578-3598-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782018616648578-3598-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.131.190:33031
--webserver_interface=127.3.131.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782018616648578-3598-0/krb5kdc/oryx_127.3.131.190.keytab
--principal=oryx/127.3.131.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.131.148:42627
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.131.190:33031
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782018616648578-3598-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782018616648578-3598-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782018616648578-3598-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 05:10:51.202149 11091 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 05:10:51.202385 11091 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 05:10:51.202409 11091 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 05:10:51.203577 11091 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 05:10:51.203612 11091 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260621 05:10:51.203626 11091 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 05:10:51.203639 11091 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 05:10:51.203652 11091 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 05:10:51.203665 11091 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 05:10:51.205312 11091 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.131.148:42627
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782018616648578-3598-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782018616648578-3598-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.131.190:33031
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782018616648578-3598-0/krb5kdc/oryx_127.3.131.190.keytab
--principal=oryx/127.3.131.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.131.190:33031
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782018616648578-3598-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.131.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11091
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782018616648578-3598-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 21 Jun 2026 04:16:16 UTC on ccycloud.quasar-xycaha.root.comops.site
build id 4457881
I20260621 05:10:51.205534 11091 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 05:10:51.205713 11091 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 05:10:51.211395 11099 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:10:51.211396 11096 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:10:51.211436 11091 server_base.cc:1034] running on GCE node
W20260621 05:10:51.211396 11097 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:10:51.212028 11091 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 05:10:51.212375 11091 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 05:10:51.213567 11091 hybrid_clock.cc:648] HybridClock initialized: now 1782018651213581 us; error 61 us; skew 500 ppm
Jun 21 05:10:51 dist-test-slave-vrq1 krb5kdc[11078](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782018651, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.131.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 05:10:51.216441 11091 init.cc:377] Logged in from keytab as oryx/127.3.131.190@KRBTEST.COM (short username oryx)
I20260621 05:10:51.216971 11091 webserver.cc:460] Webserver started at http://127.3.131.190:38647/ using document root <none> and password file <none>
I20260621 05:10:51.217247 11091 fs_manager.cc:359] Metadata directory not provided
I20260621 05:10:51.217296 11091 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 05:10:51.217448 11091 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 05:10:51.218631 11091 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782018616648578-3598-0/minicluster-data/master-0/data/instance:
uuid: "65139302364d411cb10a6e712d783f40"
format_stamp: "Formatted at 2026-06-21 05:10:51 on dist-test-slave-vrq1"
server_key: "ef6148c9db24c3637069acb6c709f345"
server_key_iv: "0b6daf1bdad8439b2e734d7b137d96be"
server_key_version: "encryptionkey@0"
I20260621 05:10:51.219033 11091 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782018616648578-3598-0/minicluster-data/master-0/wal/instance:
uuid: "65139302364d411cb10a6e712d783f40"
format_stamp: "Formatted at 2026-06-21 05:10:51 on dist-test-slave-vrq1"
server_key: "ef6148c9db24c3637069acb6c709f345"
server_key_iv: "0b6daf1bdad8439b2e734d7b137d96be"
server_key_version: "encryptionkey@0"
I20260621 05:10:51.220587 11091 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260621 05:10:51.221541 11106 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 05:10:51.221801 11091 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260621 05:10:51.221907 11091 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782018616648578-3598-0/minicluster-data/master-0/data,/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782018616648578-3598-0/minicluster-data/master-0/wal
uuid: "65139302364d411cb10a6e712d783f40"
format_stamp: "Formatted at 2026-06-21 05:10:51 on dist-test-slave-vrq1"
server_key: "ef6148c9db24c3637069acb6c709f345"
server_key_iv: "0b6daf1bdad8439b2e734d7b137d96be"
server_key_version: "encryptionkey@0"
I20260621 05:10:51.221971 11091 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782018616648578-3598-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782018616648578-3598-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782018616648578-3598-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 05:10:51.239380 11091 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 05:10:51.241531 11091 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 05:10:51.241649 11091 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 05:10:51.246922 11091 rpc_server.cc:307] RPC server started. Bound to: 127.3.131.190:33031
I20260621 05:10:51.247071 11168 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.131.190:33031 every 8 connection(s)
I20260621 05:10:51.247447 11091 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782018616648578-3598-0/minicluster-data/master-0/data/info.pb
I20260621 05:10:51.248325 11169 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 05:10:51.250737 11169 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 65139302364d411cb10a6e712d783f40: Bootstrap starting.
I20260621 05:10:51.251432 11169 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 65139302364d411cb10a6e712d783f40: Neither blocks nor log segments found. Creating new log.
I20260621 05:10:51.251756 11169 log.cc:824] T 00000000000000000000000000000000 P 65139302364d411cb10a6e712d783f40: Log is configured to *not* fsync() on all Append() calls
I20260621 05:10:51.252592 11169 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 65139302364d411cb10a6e712d783f40: No bootstrap required, opened a new log
I20260621 05:10:51.254882 11169 raft_consensus.cc:348] T 00000000000000000000000000000000 P 65139302364d411cb10a6e712d783f40 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "65139302364d411cb10a6e712d783f40" member_type: VOTER last_known_addr { host: "127.3.131.190" port: 33031 } }
I20260621 05:10:51.255045 11169 raft_consensus.cc:374] T 00000000000000000000000000000000 P 65139302364d411cb10a6e712d783f40 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 05:10:51.255069 11169 raft_consensus.cc:729] T 00000000000000000000000000000000 P 65139302364d411cb10a6e712d783f40 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 65139302364d411cb10a6e712d783f40, State: Initialized, Role: FOLLOWER
I20260621 05:10:51.255278 11169 consensus_queue.cc:260] T 00000000000000000000000000000000 P 65139302364d411cb10a6e712d783f40 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "65139302364d411cb10a6e712d783f40" member_type: VOTER last_known_addr { host: "127.3.131.190" port: 33031 } }
I20260621 05:10:51.255362 11169 raft_consensus.cc:388] T 00000000000000000000000000000000 P 65139302364d411cb10a6e712d783f40 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260621 05:10:51.255386 11169 raft_consensus.cc:482] T 00000000000000000000000000000000 P 65139302364d411cb10a6e712d783f40 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260621 05:10:51.255451 11169 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 65139302364d411cb10a6e712d783f40 [term 0 FOLLOWER]: Advancing to term 1
I20260621 05:10:51.255714 3598 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWZOx2_/build/release/bin/kudu as pid 11091
I20260621 05:10:51.255820 3598 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782018616648578-3598-0/minicluster-data/master-0/wal/instance
I20260621 05:10:51.256016 3598 external_mini_cluster.cc:1428] Setting key c54b62e3f10ee9495a43869ced23d96f
I20260621 05:10:51.256242 11169 raft_consensus.cc:504] T 00000000000000000000000000000000 P 65139302364d411cb10a6e712d783f40 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "65139302364d411cb10a6e712d783f40" member_type: VOTER last_known_addr { host: "127.3.131.190" port: 33031 } }
I20260621 05:10:51.256366 11169 leader_election.cc:302] T 00000000000000000000000000000000 P 65139302364d411cb10a6e712d783f40 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 65139302364d411cb10a6e712d783f40; no voters:
I20260621 05:10:51.256568 11169 leader_election.cc:288] T 00000000000000000000000000000000 P 65139302364d411cb10a6e712d783f40 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260621 05:10:51.256857 11172 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 65139302364d411cb10a6e712d783f40 [term 1 FOLLOWER]: Leader election won for term 1
I20260621 05:10:51.256892 11169 sys_catalog.cc:564] T 00000000000000000000000000000000 P 65139302364d411cb10a6e712d783f40 [sys.catalog]: configured and running, proceeding with master startup.
I20260621 05:10:51.257043 11172 raft_consensus.cc:686] T 00000000000000000000000000000000 P 65139302364d411cb10a6e712d783f40 [term 1 LEADER]: Becoming Leader. State: Replica: 65139302364d411cb10a6e712d783f40, State: Running, Role: LEADER
I20260621 05:10:51.257180 11172 consensus_queue.cc:237] T 00000000000000000000000000000000 P 65139302364d411cb10a6e712d783f40 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "65139302364d411cb10a6e712d783f40" member_type: VOTER last_known_addr { host: "127.3.131.190" port: 33031 } }
I20260621 05:10:51.257668 11175 sys_catalog.cc:455] T 00000000000000000000000000000000 P 65139302364d411cb10a6e712d783f40 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "65139302364d411cb10a6e712d783f40" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "65139302364d411cb10a6e712d783f40" member_type: VOTER last_known_addr { host: "127.3.131.190" port: 33031 } } }
I20260621 05:10:51.257768 11175 sys_catalog.cc:458] T 00000000000000000000000000000000 P 65139302364d411cb10a6e712d783f40 [sys.catalog]: This master's current role is: LEADER
I20260621 05:10:51.257679 11176 sys_catalog.cc:455] T 00000000000000000000000000000000 P 65139302364d411cb10a6e712d783f40 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 65139302364d411cb10a6e712d783f40. Latest consensus state: current_term: 1 leader_uuid: "65139302364d411cb10a6e712d783f40" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "65139302364d411cb10a6e712d783f40" member_type: VOTER last_known_addr { host: "127.3.131.190" port: 33031 } } }
I20260621 05:10:51.257874 11176 sys_catalog.cc:458] T 00000000000000000000000000000000 P 65139302364d411cb10a6e712d783f40 [sys.catalog]: This master's current role is: LEADER
I20260621 05:10:51.258069 11183 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260621 05:10:51.258601 11183 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260621 05:10:51.260309 11183 catalog_manager.cc:1347] Generated new cluster ID: 54ad72847fe3461bb64a42853437ceb1
I20260621 05:10:51.260362 11183 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Jun 21 05:10:51 dist-test-slave-vrq1 krb5kdc[11078](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782018651, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.3.131.190@KRBTEST.COM
I20260621 05:10:51.269099 11174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:10:51.256580 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48560 (local address 127.3.131.190:33031)
0621 05:10:51.256780 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0621 05:10:51.256791 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0621 05:10:51.256828 (+ 37us) server_negotiation.cc:407] Connection header received
0621 05:10:51.256961 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:10:51.256970 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:10:51.257029 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:10:51.257175 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:10:51.258226 (+ 1051us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:10:51.259789 (+ 1563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:10:51.260867 (+ 1078us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:10:51.261548 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:10:51.261663 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:10:51.264818 (+ 3155us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:10:51.264825 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:10:51.264826 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:10:51.264841 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:10:51.267047 (+ 2206us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:10:51.267558 (+ 511us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:10:51.267562 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:10:51.267562 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:10:51.267712 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:10:51.267926 (+ 214us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:10:51.267927 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:10:51.267928 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:10:51.268306 (+ 378us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:10:51.268480 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0621 05:10:51.268567 (+ 87us) server_negotiation.cc:299] Negotiation successful
0621 05:10:51.268851 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":145,"thread_start_us":88,"threads_started":1}
I20260621 05:10:51.281400 11183 catalog_manager.cc:1370] Generated new certificate authority record
I20260621 05:10:51.281973 11183 catalog_manager.cc:1504] Loading token signing keys...
I20260621 05:10:51.287027 11183 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 65139302364d411cb10a6e712d783f40: Generated new TSK 0
I20260621 05:10:51.287300 11183 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.3.131.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.3.131.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.3.131.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782018616648578-3598-0/krb5kdc/oryx_127.3.131.129.keytab.
Entry for principal oryx/127.3.131.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782018616648578-3598-0/krb5kdc/oryx_127.3.131.129.keytab.
I20260621 05:10:51.322373 3598 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWZOx2_/build/release/bin/kudu
/tmp/dist-test-taskWZOx2_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782018616648578-3598-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782018616648578-3598-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782018616648578-3598-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782018616648578-3598-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.131.129:0
--local_ip_for_outbound_sockets=127.3.131.129
--webserver_interface=127.3.131.129
--webserver_port=0
--tserver_master_addrs=127.3.131.190:33031
--keytab_file=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782018616648578-3598-0/krb5kdc/oryx_127.3.131.129.keytab
--principal=oryx/127.3.131.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.131.148:42627
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782018616648578-3598-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782018616648578-3598-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782018616648578-3598-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 05:10:51.411401 11195 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 05:10:51.411621 11195 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 05:10:51.411649 11195 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 05:10:51.412809 11195 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 05:10:51.412851 11195 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 05:10:51.412873 11195 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260621 05:10:51.412890 11195 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.131.129
I20260621 05:10:51.414395 11195 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.131.148:42627
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782018616648578-3598-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782018616648578-3598-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782018616648578-3598-0/krb5kdc/oryx_127.3.131.129.keytab
--principal=oryx/127.3.131.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.131.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782018616648578-3598-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.131.129
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.3.131.190:33031
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11195
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.131.129
--log_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782018616648578-3598-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 21 Jun 2026 04:16:16 UTC on ccycloud.quasar-xycaha.root.comops.site
build id 4457881
I20260621 05:10:51.414595 11195 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 05:10:51.414785 11195 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 05:10:51.420101 11200 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:10:51.420117 11203 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:10:51.420349 11201 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:10:51.420354 11195 server_base.cc:1034] running on GCE node
I20260621 05:10:51.420637 11195 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 05:10:51.420907 11195 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 05:10:51.422083 11195 hybrid_clock.cc:648] HybridClock initialized: now 1782018651422076 us; error 37 us; skew 500 ppm
Jun 21 05:10:51 dist-test-slave-vrq1 krb5kdc[11078](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782018651, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.131.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 05:10:51.424849 11195 init.cc:377] Logged in from keytab as oryx/127.3.131.129@KRBTEST.COM (short username oryx)
I20260621 05:10:51.425499 11195 webserver.cc:460] Webserver started at http://127.3.131.129:38825/ using document root <none> and password file <none>
I20260621 05:10:51.425732 11195 fs_manager.cc:359] Metadata directory not provided
I20260621 05:10:51.425776 11195 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 05:10:51.425868 11195 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 05:10:51.426967 11195 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782018616648578-3598-0/minicluster-data/ts-0/data/instance:
uuid: "6fa73262ba394487a5a54708f5de7075"
format_stamp: "Formatted at 2026-06-21 05:10:51 on dist-test-slave-vrq1"
server_key: "b59a761771c30036e891f79bac61a850"
server_key_iv: "5c5edbe3027e18e5484195154d2fb814"
server_key_version: "encryptionkey@0"
I20260621 05:10:51.427407 11195 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782018616648578-3598-0/minicluster-data/ts-0/wal/instance:
uuid: "6fa73262ba394487a5a54708f5de7075"
format_stamp: "Formatted at 2026-06-21 05:10:51 on dist-test-slave-vrq1"
server_key: "b59a761771c30036e891f79bac61a850"
server_key_iv: "5c5edbe3027e18e5484195154d2fb814"
server_key_version: "encryptionkey@0"
I20260621 05:10:51.428742 11195 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260621 05:10:51.429536 11210 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 05:10:51.429733 11195 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260621 05:10:51.429853 11195 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782018616648578-3598-0/minicluster-data/ts-0/data,/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782018616648578-3598-0/minicluster-data/ts-0/wal
uuid: "6fa73262ba394487a5a54708f5de7075"
format_stamp: "Formatted at 2026-06-21 05:10:51 on dist-test-slave-vrq1"
server_key: "b59a761771c30036e891f79bac61a850"
server_key_iv: "5c5edbe3027e18e5484195154d2fb814"
server_key_version: "encryptionkey@0"
I20260621 05:10:51.429927 11195 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782018616648578-3598-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782018616648578-3598-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782018616648578-3598-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 05:10:51.443518 11195 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 05:10:51.446168 11195 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 05:10:51.446298 11195 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 05:10:51.446805 11195 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 05:10:51.446885 11195 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 05:10:51.446939 11195 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 05:10:51.446996 11195 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 05:10:51.453908 11195 rpc_server.cc:307] RPC server started. Bound to: 127.3.131.129:38555
I20260621 05:10:51.454073 11325 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.131.129:38555 every 8 connection(s)
I20260621 05:10:51.454335 11195 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782018616648578-3598-0/minicluster-data/ts-0/data/info.pb
Jun 21 05:10:51 dist-test-slave-vrq1 krb5kdc[11078](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782018651, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.131.129@KRBTEST.COM for oryx/127.3.131.190@KRBTEST.COM
I20260621 05:10:51.458088 3598 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWZOx2_/build/release/bin/kudu as pid 11195
I20260621 05:10:51.458261 3598 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782018616648578-3598-0/minicluster-data/ts-0/wal/instance
I20260621 05:10:51.458451 3598 external_mini_cluster.cc:1428] Setting key 9fb05c3d5be92a1cc2bbddb1864b827a
I20260621 05:10:51.459028 11174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:10:51.447528 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.131.129:40515 (local address 127.3.131.190:33031)
0621 05:10:51.447608 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0621 05:10:51.447610 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:10:51.447812 (+ 202us) server_negotiation.cc:407] Connection header received
0621 05:10:51.448334 (+ 522us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:10:51.448336 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:10:51.448351 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:10:51.448434 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:10:51.449376 (+ 942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:10:51.450271 (+ 895us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:10:51.452153 (+ 1882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:10:51.452557 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:10:51.452640 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:10:51.455595 (+ 2955us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:10:51.455596 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:10:51.455596 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:10:51.455618 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:10:51.457221 (+ 1603us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:10:51.457718 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:10:51.457719 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:10:51.457719 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:10:51.457791 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:10:51.458212 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:10:51.458213 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:10:51.458213 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:10:51.458390 (+ 177us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:10:51.458472 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0621 05:10:51.458831 (+ 359us) server_negotiation.cc:299] Negotiation successful
0621 05:10:51.458929 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260621 05:10:51.459306 11219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:10:51.447490 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.131.190:33031 (local address 127.3.131.129:40515)
0621 05:10:51.447686 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0621 05:10:51.447728 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0621 05:10:51.448219 (+ 491us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:10:51.448495 (+ 276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:10:51.448498 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:10:51.448931 (+ 433us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:10:51.449245 (+ 314us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:10:51.449251 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:10:51.450398 (+ 1147us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:10:51.450399 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:10:51.452046 (+ 1647us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:10:51.452050 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:10:51.452683 (+ 633us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:10:51.452685 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:10:51.452868 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:10:51.453645 (+ 777us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 05:10:51.453655 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:10:51.455486 (+ 1831us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 05:10:51.457361 (+ 1875us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:10:51.457369 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:10:51.457371 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:10:51.457627 (+ 256us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:10:51.457947 (+ 320us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:10:51.457948 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:10:51.457948 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:10:51.458120 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:10:51.458478 (+ 358us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 05:10:51.458485 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 05:10:51.458688 (+ 203us) client_negotiation.cc:769] Sending connection context
0621 05:10:51.458805 (+ 117us) client_negotiation.cc:240] Negotiation successful
0621 05:10:51.459008 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":156,"thread_start_us":94,"threads_started":1}
I20260621 05:10:51.459854 11326 heartbeater.cc:347] Connected to a master server at 127.3.131.190:33031
I20260621 05:10:51.459939 11326 heartbeater.cc:464] Registering TS with master...
I20260621 05:10:51.460093 11326 heartbeater.cc:511] Master 127.3.131.190:33031 requested a full tablet report, sending...
I20260621 05:10:51.460568 11122 ts_manager.cc:194] Registered new tserver with Master: 6fa73262ba394487a5a54708f5de7075 (127.3.131.129:38555)
I20260621 05:10:51.461225 11122 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.3.131.129@KRBTEST.COM'} at 127.3.131.129:40515
WARNING: no policy specified for oryx/127.3.131.130@KRBTEST.COM; defaulting to no policy
I20260621 05:10:51.468822 11333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:10:51.461588 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.131.190:33031 (local address 127.3.131.129:37277)
0621 05:10:51.461787 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0621 05:10:51.461799 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0621 05:10:51.461885 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:10:51.462070 (+ 185us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:10:51.462070 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:10:51.462302 (+ 232us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:10:51.462363 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:10:51.462365 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:10:51.463356 (+ 991us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:10:51.463357 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:10:51.463980 (+ 623us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:10:51.463983 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:10:51.464437 (+ 454us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:10:51.464439 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:10:51.464527 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:10:51.465208 (+ 681us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 05:10:51.465215 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:10:51.465729 (+ 514us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 05:10:51.467525 (+ 1796us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:10:51.467527 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:10:51.467528 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:10:51.467751 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:10:51.468098 (+ 347us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:10:51.468099 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:10:51.468099 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:10:51.468177 (+ 78us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:10:51.468483 (+ 306us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 05:10:51.468484 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 05:10:51.468544 (+ 60us) client_negotiation.cc:769] Sending connection context
0621 05:10:51.468619 (+ 75us) client_negotiation.cc:240] Negotiation successful
0621 05:10:51.468730 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":162,"thread_start_us":90,"threads_started":1}
I20260621 05:10:51.468881 11174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:10:51.461663 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.131.129:37277 (local address 127.3.131.190:33031)
0621 05:10:51.461710 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0621 05:10:51.461712 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:10:51.461855 (+ 143us) server_negotiation.cc:407] Connection header received
0621 05:10:51.461962 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:10:51.461963 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:10:51.461974 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:10:51.462003 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:10:51.462446 (+ 443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:10:51.463249 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:10:51.464077 (+ 828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:10:51.464355 (+ 278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:10:51.464413 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:10:51.465833 (+ 1420us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:10:51.465835 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:10:51.465835 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:10:51.465842 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:10:51.467440 (+ 1598us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:10:51.467855 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:10:51.467857 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:10:51.467857 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:10:51.467914 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:10:51.468268 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:10:51.468269 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:10:51.468269 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:10:51.468391 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:10:51.468460 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0621 05:10:51.468686 (+ 226us) server_negotiation.cc:299] Negotiation successful
0621 05:10:51.468776 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.3.131.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.3.131.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782018616648578-3598-0/krb5kdc/oryx_127.3.131.130.keytab.
Entry for principal oryx/127.3.131.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782018616648578-3598-0/krb5kdc/oryx_127.3.131.130.keytab.
I20260621 05:10:51.487905 3598 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWZOx2_/build/release/bin/kudu
/tmp/dist-test-taskWZOx2_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782018616648578-3598-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782018616648578-3598-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782018616648578-3598-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782018616648578-3598-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.131.130:0
--local_ip_for_outbound_sockets=127.3.131.130
--webserver_interface=127.3.131.130
--webserver_port=0
--tserver_master_addrs=127.3.131.190:33031
--keytab_file=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782018616648578-3598-0/krb5kdc/oryx_127.3.131.130.keytab
--principal=oryx/127.3.131.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.131.148:42627
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782018616648578-3598-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782018616648578-3598-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782018616648578-3598-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 05:10:51.578078 11335 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 05:10:51.578330 11335 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 05:10:51.578361 11335 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 05:10:51.579902 11335 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 05:10:51.579981 11335 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 05:10:51.580008 11335 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260621 05:10:51.580026 11335 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.131.130
I20260621 05:10:51.581743 11335 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.131.148:42627
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782018616648578-3598-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782018616648578-3598-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782018616648578-3598-0/krb5kdc/oryx_127.3.131.130.keytab
--principal=oryx/127.3.131.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.131.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782018616648578-3598-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.131.130
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.3.131.190:33031
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11335
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.131.130
--log_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782018616648578-3598-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 21 Jun 2026 04:16:16 UTC on ccycloud.quasar-xycaha.root.comops.site
build id 4457881
I20260621 05:10:51.581966 11335 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 05:10:51.582165 11335 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 05:10:51.588660 11343 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:10:51.588636 11340 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:10:51.588636 11341 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:10:51.589094 11335 server_base.cc:1034] running on GCE node
I20260621 05:10:51.589315 11335 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 05:10:51.589808 11335 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 05:10:51.591048 11335 hybrid_clock.cc:648] HybridClock initialized: now 1782018651590985 us; error 80 us; skew 500 ppm
Jun 21 05:10:51 dist-test-slave-vrq1 krb5kdc[11078](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782018651, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.131.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 05:10:51.594514 11335 init.cc:377] Logged in from keytab as oryx/127.3.131.130@KRBTEST.COM (short username oryx)
I20260621 05:10:51.595122 11335 webserver.cc:460] Webserver started at http://127.3.131.130:39259/ using document root <none> and password file <none>
I20260621 05:10:51.595413 11335 fs_manager.cc:359] Metadata directory not provided
I20260621 05:10:51.595487 11335 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 05:10:51.595652 11335 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 05:10:51.596853 11335 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782018616648578-3598-0/minicluster-data/ts-1/data/instance:
uuid: "eceee696a5ee4011b1264da66419eae9"
format_stamp: "Formatted at 2026-06-21 05:10:51 on dist-test-slave-vrq1"
server_key: "e6e9841136659d25f992f47d0596b1bc"
server_key_iv: "db284d79b022a7f7b4ad2485247966ad"
server_key_version: "encryptionkey@0"
I20260621 05:10:51.597265 11335 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782018616648578-3598-0/minicluster-data/ts-1/wal/instance:
uuid: "eceee696a5ee4011b1264da66419eae9"
format_stamp: "Formatted at 2026-06-21 05:10:51 on dist-test-slave-vrq1"
server_key: "e6e9841136659d25f992f47d0596b1bc"
server_key_iv: "db284d79b022a7f7b4ad2485247966ad"
server_key_version: "encryptionkey@0"
I20260621 05:10:51.598711 11335 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260621 05:10:51.599692 11350 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 05:10:51.599920 11335 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260621 05:10:51.600095 11335 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782018616648578-3598-0/minicluster-data/ts-1/data,/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782018616648578-3598-0/minicluster-data/ts-1/wal
uuid: "eceee696a5ee4011b1264da66419eae9"
format_stamp: "Formatted at 2026-06-21 05:10:51 on dist-test-slave-vrq1"
server_key: "e6e9841136659d25f992f47d0596b1bc"
server_key_iv: "db284d79b022a7f7b4ad2485247966ad"
server_key_version: "encryptionkey@0"
I20260621 05:10:51.600173 11335 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782018616648578-3598-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782018616648578-3598-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782018616648578-3598-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 05:10:51.611997 11335 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 05:10:51.614398 11335 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 05:10:51.614578 11335 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 05:10:51.615216 11335 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 05:10:51.615273 11335 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 05:10:51.615300 11335 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 05:10:51.615330 11335 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 05:10:51.622668 11335 rpc_server.cc:307] RPC server started. Bound to: 127.3.131.130:36509
I20260621 05:10:51.622843 11465 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.131.130:36509 every 8 connection(s)
I20260621 05:10:51.623225 11335 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782018616648578-3598-0/minicluster-data/ts-1/data/info.pb
Jun 21 05:10:51 dist-test-slave-vrq1 krb5kdc[11078](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782018651, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.131.130@KRBTEST.COM for oryx/127.3.131.190@KRBTEST.COM
I20260621 05:10:51.624135 3598 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWZOx2_/build/release/bin/kudu as pid 11335
I20260621 05:10:51.624270 3598 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782018616648578-3598-0/minicluster-data/ts-1/wal/instance
I20260621 05:10:51.624467 3598 external_mini_cluster.cc:1428] Setting key ccc3ae3b1c4fb70fd3b8de572fbc9b96
I20260621 05:10:51.628136 11174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:10:51.615842 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.131.130:40597 (local address 127.3.131.190:33031)
0621 05:10:51.615894 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0621 05:10:51.615896 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:10:51.616169 (+ 273us) server_negotiation.cc:407] Connection header received
0621 05:10:51.616753 (+ 584us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:10:51.616754 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:10:51.616768 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:10:51.616814 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:10:51.617714 (+ 900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:10:51.618536 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:10:51.620637 (+ 2101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:10:51.620978 (+ 341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:10:51.621105 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:10:51.624423 (+ 3318us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:10:51.624425 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:10:51.624425 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:10:51.624434 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:10:51.626244 (+ 1810us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:10:51.626900 (+ 656us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:10:51.626901 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:10:51.626901 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:10:51.626969 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:10:51.627350 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:10:51.627351 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:10:51.627351 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:10:51.627479 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:10:51.627538 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0621 05:10:51.627968 (+ 430us) server_negotiation.cc:299] Negotiation successful
0621 05:10:51.628053 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260621 05:10:51.628412 11359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:10:51.615839 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.131.190:33031 (local address 127.3.131.130:40597)
0621 05:10:51.616039 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0621 05:10:51.616078 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0621 05:10:51.616631 (+ 553us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:10:51.616922 (+ 291us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:10:51.616927 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:10:51.617307 (+ 380us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:10:51.617624 (+ 317us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:10:51.617633 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:10:51.618628 (+ 995us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:10:51.618629 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:10:51.620394 (+ 1765us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:10:51.620400 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:10:51.621172 (+ 772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:10:51.621174 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:10:51.621348 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:10:51.622217 (+ 869us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 05:10:51.622227 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:10:51.624265 (+ 2038us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 05:10:51.626337 (+ 2072us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:10:51.626349 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:10:51.626352 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:10:51.626765 (+ 413us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:10:51.627050 (+ 285us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:10:51.627051 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:10:51.627052 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:10:51.627282 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:10:51.627553 (+ 271us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 05:10:51.627561 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 05:10:51.627768 (+ 207us) client_negotiation.cc:769] Sending connection context
0621 05:10:51.627889 (+ 121us) client_negotiation.cc:240] Negotiation successful
0621 05:10:51.628108 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":89,"threads_started":1}
I20260621 05:10:51.629108 11466 heartbeater.cc:347] Connected to a master server at 127.3.131.190:33031
I20260621 05:10:51.629220 11466 heartbeater.cc:464] Registering TS with master...
I20260621 05:10:51.629427 11466 heartbeater.cc:511] Master 127.3.131.190:33031 requested a full tablet report, sending...
I20260621 05:10:51.629869 11123 ts_manager.cc:194] Registered new tserver with Master: eceee696a5ee4011b1264da66419eae9 (127.3.131.130:36509)
I20260621 05:10:51.630285 11123 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.3.131.130@KRBTEST.COM'} at 127.3.131.130:40597
WARNING: no policy specified for oryx/127.3.131.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.3.131.131@KRBTEST.COM" created.
I20260621 05:10:51.638558 11174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:10:51.630503 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.131.130:42911 (local address 127.3.131.190:33031)
0621 05:10:51.630548 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0621 05:10:51.630550 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:10:51.631417 (+ 867us) server_negotiation.cc:407] Connection header received
0621 05:10:51.631486 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:10:51.631487 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:10:51.631496 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:10:51.631546 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:10:51.631972 (+ 426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:10:51.632480 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:10:51.633659 (+ 1179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:10:51.633970 (+ 311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:10:51.634031 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:10:51.635243 (+ 1212us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:10:51.635245 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:10:51.635246 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:10:51.635260 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:10:51.636777 (+ 1517us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:10:51.637239 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:10:51.637240 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:10:51.637240 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:10:51.637302 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:10:51.637739 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:10:51.637740 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:10:51.637741 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:10:51.638070 (+ 329us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:10:51.638167 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0621 05:10:51.638400 (+ 233us) server_negotiation.cc:299] Negotiation successful
0621 05:10:51.638481 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260621 05:10:51.638674 11473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:10:51.630558 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.131.190:33031 (local address 127.3.131.130:42911)
0621 05:10:51.631324 (+ 766us) negotiation.cc:107] Waiting for socket to connect
0621 05:10:51.631337 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0621 05:10:51.631414 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:10:51.631557 (+ 143us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:10:51.631557 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:10:51.631805 (+ 248us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:10:51.631886 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:10:51.631888 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:10:51.632594 (+ 706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:10:51.632595 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:10:51.633545 (+ 950us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:10:51.633548 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:10:51.634054 (+ 506us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:10:51.634055 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:10:51.634134 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:10:51.634698 (+ 564us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 05:10:51.634704 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:10:51.635099 (+ 395us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 05:10:51.636892 (+ 1793us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:10:51.636893 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:10:51.636894 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:10:51.637160 (+ 266us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:10:51.637392 (+ 232us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:10:51.637392 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:10:51.637393 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:10:51.637459 (+ 66us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:10:51.638177 (+ 718us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 05:10:51.638178 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 05:10:51.638237 (+ 59us) client_negotiation.cc:769] Sending connection context
0621 05:10:51.638335 (+ 98us) client_negotiation.cc:240] Negotiation successful
0621 05:10:51.638515 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":731,"thread_start_us":83,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.3.131.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782018616648578-3598-0/krb5kdc/oryx_127.3.131.131.keytab.
Entry for principal oryx/127.3.131.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782018616648578-3598-0/krb5kdc/oryx_127.3.131.131.keytab.
I20260621 05:10:51.658277 3598 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWZOx2_/build/release/bin/kudu
/tmp/dist-test-taskWZOx2_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782018616648578-3598-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782018616648578-3598-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782018616648578-3598-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782018616648578-3598-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.131.131:0
--local_ip_for_outbound_sockets=127.3.131.131
--webserver_interface=127.3.131.131
--webserver_port=0
--tserver_master_addrs=127.3.131.190:33031
--keytab_file=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782018616648578-3598-0/krb5kdc/oryx_127.3.131.131.keytab
--principal=oryx/127.3.131.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.131.148:42627
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782018616648578-3598-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782018616648578-3598-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782018616648578-3598-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 05:10:51.742872 11475 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 05:10:51.743137 11475 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 05:10:51.743181 11475 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 05:10:51.744428 11475 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 05:10:51.744506 11475 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 05:10:51.744554 11475 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260621 05:10:51.744612 11475 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.131.131
I20260621 05:10:51.746202 11475 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.131.148:42627
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782018616648578-3598-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782018616648578-3598-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782018616648578-3598-0/krb5kdc/oryx_127.3.131.131.keytab
--principal=oryx/127.3.131.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.131.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782018616648578-3598-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.131.131
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.3.131.190:33031
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11475
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.131.131
--log_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782018616648578-3598-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 21 Jun 2026 04:16:16 UTC on ccycloud.quasar-xycaha.root.comops.site
build id 4457881
I20260621 05:10:51.746481 11475 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 05:10:51.746728 11475 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 05:10:51.752525 11481 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:10:51.752696 11475 server_base.cc:1034] running on GCE node
W20260621 05:10:51.752525 11483 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:10:51.752562 11480 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:10:51.753041 11475 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 05:10:51.753301 11475 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 05:10:51.754482 11475 hybrid_clock.cc:648] HybridClock initialized: now 1782018651754444 us; error 64 us; skew 500 ppm
Jun 21 05:10:51 dist-test-slave-vrq1 krb5kdc[11078](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782018651, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.131.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 05:10:51.757881 11475 init.cc:377] Logged in from keytab as oryx/127.3.131.131@KRBTEST.COM (short username oryx)
I20260621 05:10:51.758433 11475 webserver.cc:460] Webserver started at http://127.3.131.131:40809/ using document root <none> and password file <none>
I20260621 05:10:51.758710 11475 fs_manager.cc:359] Metadata directory not provided
I20260621 05:10:51.758785 11475 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 05:10:51.758934 11475 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 05:10:51.760138 11475 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782018616648578-3598-0/minicluster-data/ts-2/data/instance:
uuid: "017e3652ab544ab08232482357ea8d48"
format_stamp: "Formatted at 2026-06-21 05:10:51 on dist-test-slave-vrq1"
server_key: "3f6343f821e2aaa92e300239d8d59d11"
server_key_iv: "3526218b1c5dd41f4cdaa02b79f03fd3"
server_key_version: "encryptionkey@0"
I20260621 05:10:51.760507 11475 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782018616648578-3598-0/minicluster-data/ts-2/wal/instance:
uuid: "017e3652ab544ab08232482357ea8d48"
format_stamp: "Formatted at 2026-06-21 05:10:51 on dist-test-slave-vrq1"
server_key: "3f6343f821e2aaa92e300239d8d59d11"
server_key_iv: "3526218b1c5dd41f4cdaa02b79f03fd3"
server_key_version: "encryptionkey@0"
I20260621 05:10:51.761991 11475 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260621 05:10:51.762854 11490 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 05:10:51.763079 11475 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260621 05:10:51.763206 11475 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782018616648578-3598-0/minicluster-data/ts-2/data,/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782018616648578-3598-0/minicluster-data/ts-2/wal
uuid: "017e3652ab544ab08232482357ea8d48"
format_stamp: "Formatted at 2026-06-21 05:10:51 on dist-test-slave-vrq1"
server_key: "3f6343f821e2aaa92e300239d8d59d11"
server_key_iv: "3526218b1c5dd41f4cdaa02b79f03fd3"
server_key_version: "encryptionkey@0"
I20260621 05:10:51.763283 11475 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782018616648578-3598-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782018616648578-3598-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782018616648578-3598-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 05:10:51.772152 11475 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 05:10:51.774453 11475 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 05:10:51.774593 11475 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 05:10:51.775102 11475 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 05:10:51.775144 11475 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 05:10:51.775236 11475 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 05:10:51.775270 11475 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 05:10:51.781988 11475 rpc_server.cc:307] RPC server started. Bound to: 127.3.131.131:46349
I20260621 05:10:51.782150 11605 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.131.131:46349 every 8 connection(s)
I20260621 05:10:51.782317 11475 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782018616648578-3598-0/minicluster-data/ts-2/data/info.pb
Jun 21 05:10:51 dist-test-slave-vrq1 krb5kdc[11078](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782018651, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.131.131@KRBTEST.COM for oryx/127.3.131.190@KRBTEST.COM
I20260621 05:10:51.783915 3598 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWZOx2_/build/release/bin/kudu as pid 11475
I20260621 05:10:51.784013 3598 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782018616648578-3598-0/minicluster-data/ts-2/wal/instance
I20260621 05:10:51.784251 3598 external_mini_cluster.cc:1428] Setting key 154969d20bc88083041a2813f2ffb73b
I20260621 05:10:51.786929 11174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:10:51.775708 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.131.131:33673 (local address 127.3.131.190:33031)
0621 05:10:51.775790 (+ 82us) server_negotiation.cc:206] Beginning negotiation
0621 05:10:51.775793 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0621 05:10:51.776024 (+ 231us) server_negotiation.cc:407] Connection header received
0621 05:10:51.776630 (+ 606us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:10:51.776632 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:10:51.776646 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:10:51.776740 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:10:51.777647 (+ 907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:10:51.778499 (+ 852us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:10:51.780291 (+ 1792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:10:51.780625 (+ 334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:10:51.780725 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:10:51.783639 (+ 2914us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:10:51.783640 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:10:51.783641 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:10:51.783649 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:10:51.785461 (+ 1812us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:10:51.785881 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:10:51.785882 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:10:51.785882 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:10:51.785948 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:10:51.786234 (+ 286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:10:51.786235 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:10:51.786235 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:10:51.786361 (+ 126us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:10:51.786433 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0621 05:10:51.786729 (+ 296us) server_negotiation.cc:299] Negotiation successful
0621 05:10:51.786841 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260621 05:10:51.787138 11499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:10:51.775702 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.131.190:33031 (local address 127.3.131.131:33673)
0621 05:10:51.775883 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0621 05:10:51.775923 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0621 05:10:51.776499 (+ 576us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:10:51.776826 (+ 327us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:10:51.776830 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:10:51.777223 (+ 393us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:10:51.777530 (+ 307us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:10:51.777537 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:10:51.778612 (+ 1075us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:10:51.778613 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:10:51.780174 (+ 1561us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:10:51.780178 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:10:51.780732 (+ 554us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:10:51.780733 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:10:51.780901 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:10:51.781638 (+ 737us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 05:10:51.781647 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:10:51.783530 (+ 1883us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 05:10:51.785540 (+ 2010us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:10:51.785548 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:10:51.785550 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:10:51.785790 (+ 240us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:10:51.786018 (+ 228us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:10:51.786019 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:10:51.786019 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:10:51.786137 (+ 118us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:10:51.786436 (+ 299us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 05:10:51.786443 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 05:10:51.786587 (+ 144us) client_negotiation.cc:769] Sending connection context
0621 05:10:51.786705 (+ 118us) client_negotiation.cc:240] Negotiation successful
0621 05:10:51.786878 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":117,"threads_started":1}
I20260621 05:10:51.787806 11606 heartbeater.cc:347] Connected to a master server at 127.3.131.190:33031
I20260621 05:10:51.787886 11606 heartbeater.cc:464] Registering TS with master...
I20260621 05:10:51.788079 11606 heartbeater.cc:511] Master 127.3.131.190:33031 requested a full tablet report, sending...
I20260621 05:10:51.788455 11123 ts_manager.cc:194] Registered new tserver with Master: 017e3652ab544ab08232482357ea8d48 (127.3.131.131:46349)
I20260621 05:10:51.788975 11123 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.3.131.131@KRBTEST.COM'} at 127.3.131.131:33673
I20260621 05:10:51.794746 11174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:10:51.788886 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.131.131:54147 (local address 127.3.131.190:33031)
0621 05:10:51.788928 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0621 05:10:51.788930 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:10:51.789007 (+ 77us) server_negotiation.cc:407] Connection header received
0621 05:10:51.789066 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:10:51.789067 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:10:51.789077 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:10:51.789146 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:10:51.789542 (+ 396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:10:51.790017 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:10:51.790748 (+ 731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:10:51.790949 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:10:51.791033 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:10:51.792060 (+ 1027us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:10:51.792061 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:10:51.792062 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:10:51.792069 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:10:51.793469 (+ 1400us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:10:51.793865 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:10:51.793866 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:10:51.793867 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:10:51.793933 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:10:51.794166 (+ 233us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:10:51.794166 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:10:51.794167 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:10:51.794302 (+ 135us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:10:51.794382 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0621 05:10:51.794560 (+ 178us) server_negotiation.cc:299] Negotiation successful
0621 05:10:51.794645 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260621 05:10:51.794762 11612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:10:51.788793 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.131.190:33031 (local address 127.3.131.131:54147)
0621 05:10:51.788930 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0621 05:10:51.788941 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0621 05:10:51.789012 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:10:51.789161 (+ 149us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:10:51.789161 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:10:51.789378 (+ 217us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:10:51.789449 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:10:51.789451 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:10:51.790113 (+ 662us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:10:51.790114 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:10:51.790652 (+ 538us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:10:51.790654 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:10:51.791035 (+ 381us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:10:51.791036 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:10:51.791105 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:10:51.791657 (+ 552us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 05:10:51.791662 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:10:51.791963 (+ 301us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 05:10:51.793593 (+ 1630us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:10:51.793594 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:10:51.793595 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:10:51.793785 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:10:51.794022 (+ 237us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:10:51.794023 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:10:51.794023 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:10:51.794088 (+ 65us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:10:51.794372 (+ 284us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 05:10:51.794373 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 05:10:51.794425 (+ 52us) client_negotiation.cc:769] Sending connection context
0621 05:10:51.794531 (+ 106us) client_negotiation.cc:240] Negotiation successful
0621 05:10:51.794648 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":87,"threads_started":1}
I20260621 05:10:51.796454 3598 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 21 05:10:51 dist-test-slave-vrq1 krb5kdc[11078](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782018651, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 21 05:10:51 dist-test-slave-vrq1 krb5kdc[11078](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.3.131.190@KRBTEST.COM, Server not found in Kerberos database
Jun 21 05:10:51 dist-test-slave-vrq1 krb5kdc[11078](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.3.131.190@KRBTEST.COM, Server not found in Kerberos database
W20260621 05:10:51.821033 11174 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 05:10:51.814799 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57278 (local address 127.3.131.190:33031)
0621 05:10:51.814875 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0621 05:10:51.814877 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:10:51.814886 (+ 9us) server_negotiation.cc:407] Connection header received
0621 05:10:51.814914 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:10:51.814914 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:10:51.814924 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:10:51.814976 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:10:51.815753 (+ 777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:10:51.816301 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:10:51.817147 (+ 846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:10:51.817341 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:10:51.817400 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:10:51.820908 (+ 3508us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57278: BlockingRecv error: recv got EOF from 127.0.0.1:57278 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
Jun 21 05:10:51 dist-test-slave-vrq1 krb5kdc[11078](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782018651, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.3.131.190@KRBTEST.COM
I20260621 05:10:51.829571 11174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:10:51.822034 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57290 (local address 127.3.131.190:33031)
0621 05:10:51.822160 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0621 05:10:51.822162 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:10:51.822256 (+ 94us) server_negotiation.cc:407] Connection header received
0621 05:10:51.822342 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:10:51.822343 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:10:51.822359 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:10:51.822422 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:10:51.822941 (+ 519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:10:51.823513 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:10:51.824226 (+ 713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:10:51.824482 (+ 256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:10:51.824573 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:10:51.826799 (+ 2226us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:10:51.826801 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:10:51.826801 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:10:51.826809 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:10:51.828333 (+ 1524us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:10:51.828738 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:10:51.828739 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:10:51.828739 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:10:51.828802 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:10:51.829005 (+ 203us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:10:51.829006 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:10:51.829007 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:10:51.829146 (+ 139us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:10:51.829279 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0621 05:10:51.829379 (+ 100us) server_negotiation.cc:299] Negotiation successful
0621 05:10:51.829461 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":95}
I20260621 05:10:51.831638 11123 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:57290:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260621 05:10:51.833395 11123 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260621 05:10:51.840305 11632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:10:51.836255 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38122 (local address 127.3.131.131:46349)
0621 05:10:51.836420 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0621 05:10:51.836425 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 05:10:51.836461 (+ 36us) server_negotiation.cc:407] Connection header received
0621 05:10:51.836535 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:10:51.836538 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:10:51.836554 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:10:51.836639 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 05:10:51.836899 (+ 260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:10:51.838048 (+ 1149us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:10:51.839343 (+ 1295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:10:51.839863 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:10:51.839963 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:10:51.840053 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0621 05:10:51.840157 (+ 104us) server_negotiation.cc:299] Negotiation successful
0621 05:10:51.840204 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":132,"thread_start_us":75,"threads_started":1}
I20260621 05:10:51.840296 11628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:10:51.835984 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.131.131:46349 (local address 127.0.0.1:38122)
0621 05:10:51.836409 (+ 425us) negotiation.cc:107] Waiting for socket to connect
0621 05:10:51.836417 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0621 05:10:51.836463 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:10:51.836735 (+ 272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:10:51.836739 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:10:51.836741 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 05:10:51.836815 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:10:51.836818 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:10:51.838161 (+ 1343us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:10:51.838162 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:10:51.839258 (+ 1096us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:10:51.839260 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:10:51.839967 (+ 707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:10:51.839968 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:10:51.840049 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:10:51.840065 (+ 16us) client_negotiation.cc:769] Sending connection context
0621 05:10:51.840141 (+ 76us) client_negotiation.cc:240] Negotiation successful
0621 05:10:51.840198 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":414,"thread_start_us":58,"threads_started":1}
I20260621 05:10:51.840435 11630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:10:51.835981 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.131.130:36509 (local address 127.0.0.1:35264)
0621 05:10:51.836157 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0621 05:10:51.836236 (+ 79us) client_negotiation.cc:174] Beginning negotiation
0621 05:10:51.836293 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:10:51.836790 (+ 497us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:10:51.836791 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:10:51.836794 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 05:10:51.836864 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:10:51.836866 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:10:51.837814 (+ 948us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:10:51.837815 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:10:51.839108 (+ 1293us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:10:51.839110 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:10:51.840180 (+ 1070us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:10:51.840181 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:10:51.840252 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:10:51.840260 (+ 8us) client_negotiation.cc:769] Sending connection context
0621 05:10:51.840310 (+ 50us) client_negotiation.cc:240] Negotiation successful
0621 05:10:51.840359 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":145,"mutex_wait_us":6,"thread_start_us":90,"threads_started":1}
I20260621 05:10:51.840492 11631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:10:51.836225 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41098 (local address 127.3.131.129:38555)
0621 05:10:51.836411 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0621 05:10:51.836418 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0621 05:10:51.836617 (+ 199us) server_negotiation.cc:407] Connection header received
0621 05:10:51.836683 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:10:51.836686 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:10:51.836702 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:10:51.836797 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 05:10:51.836984 (+ 187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:10:51.838072 (+ 1088us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:10:51.839209 (+ 1137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:10:51.839987 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:10:51.840061 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:10:51.840139 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0621 05:10:51.840352 (+ 213us) server_negotiation.cc:299] Negotiation successful
0621 05:10:51.840390 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":144,"thread_start_us":84,"threads_started":1}
I20260621 05:10:51.840720 11629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:10:51.835985 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.131.129:38555 (local address 127.0.0.1:41098)
0621 05:10:51.836560 (+ 575us) negotiation.cc:107] Waiting for socket to connect
0621 05:10:51.836569 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0621 05:10:51.836616 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:10:51.836863 (+ 247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:10:51.836864 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:10:51.836866 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 05:10:51.836900 (+ 34us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:10:51.836901 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:10:51.838171 (+ 1270us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:10:51.838172 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:10:51.839089 (+ 917us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:10:51.839091 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:10:51.840179 (+ 1088us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:10:51.840180 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:10:51.840252 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:10:51.840259 (+ 7us) client_negotiation.cc:769] Sending connection context
0621 05:10:51.840606 (+ 347us) client_negotiation.cc:240] Negotiation successful
0621 05:10:51.840645 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":558,"mutex_wait_us":8,"thread_start_us":90,"threads_started":1}
I20260621 05:10:51.840718 11633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:10:51.836434 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35264 (local address 127.3.131.130:36509)
0621 05:10:51.836614 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0621 05:10:51.836619 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 05:10:51.836628 (+ 9us) server_negotiation.cc:407] Connection header received
0621 05:10:51.836675 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:10:51.836678 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:10:51.836693 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:10:51.836942 (+ 249us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 05:10:51.837009 (+ 67us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:10:51.837721 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:10:51.839236 (+ 1515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:10:51.839987 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:10:51.840061 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:10:51.840139 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0621 05:10:51.840558 (+ 419us) server_negotiation.cc:299] Negotiation successful
0621 05:10:51.840620 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":143,"thread_start_us":84,"threads_started":1}
I20260621 05:10:51.841400 11400 tablet_service.cc:1467] Processing CreateTablet for tablet dd732866e60a418a8cd9b605f30c77e3 (DEFAULT_TABLE table=test-table [id=3a7e99a0a23748148205ed74756e5043]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 05:10:51.841454 11540 tablet_service.cc:1467] Processing CreateTablet for tablet dd732866e60a418a8cd9b605f30c77e3 (DEFAULT_TABLE table=test-table [id=3a7e99a0a23748148205ed74756e5043]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 05:10:51.841753 11540 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dd732866e60a418a8cd9b605f30c77e3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 05:10:51.841776 11400 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dd732866e60a418a8cd9b605f30c77e3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 05:10:51.843664 11260 tablet_service.cc:1467] Processing CreateTablet for tablet dd732866e60a418a8cd9b605f30c77e3 (DEFAULT_TABLE table=test-table [id=3a7e99a0a23748148205ed74756e5043]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 05:10:51.844025 11260 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dd732866e60a418a8cd9b605f30c77e3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 05:10:51.845007 11634 tablet_bootstrap.cc:492] T dd732866e60a418a8cd9b605f30c77e3 P eceee696a5ee4011b1264da66419eae9: Bootstrap starting.
I20260621 05:10:51.845659 11634 tablet_bootstrap.cc:654] T dd732866e60a418a8cd9b605f30c77e3 P eceee696a5ee4011b1264da66419eae9: Neither blocks nor log segments found. Creating new log.
I20260621 05:10:51.845986 11634 log.cc:824] T dd732866e60a418a8cd9b605f30c77e3 P eceee696a5ee4011b1264da66419eae9: Log is configured to *not* fsync() on all Append() calls
I20260621 05:10:51.846705 11634 tablet_bootstrap.cc:492] T dd732866e60a418a8cd9b605f30c77e3 P eceee696a5ee4011b1264da66419eae9: No bootstrap required, opened a new log
I20260621 05:10:51.846764 11634 ts_tablet_manager.cc:1397] T dd732866e60a418a8cd9b605f30c77e3 P eceee696a5ee4011b1264da66419eae9: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260621 05:10:51.847546 11635 tablet_bootstrap.cc:492] T dd732866e60a418a8cd9b605f30c77e3 P 017e3652ab544ab08232482357ea8d48: Bootstrap starting.
I20260621 05:10:51.847981 11637 tablet_bootstrap.cc:492] T dd732866e60a418a8cd9b605f30c77e3 P 6fa73262ba394487a5a54708f5de7075: Bootstrap starting.
I20260621 05:10:51.848371 11635 tablet_bootstrap.cc:654] T dd732866e60a418a8cd9b605f30c77e3 P 017e3652ab544ab08232482357ea8d48: Neither blocks nor log segments found. Creating new log.
I20260621 05:10:51.848690 11637 tablet_bootstrap.cc:654] T dd732866e60a418a8cd9b605f30c77e3 P 6fa73262ba394487a5a54708f5de7075: Neither blocks nor log segments found. Creating new log.
I20260621 05:10:51.848747 11635 log.cc:824] T dd732866e60a418a8cd9b605f30c77e3 P 017e3652ab544ab08232482357ea8d48: Log is configured to *not* fsync() on all Append() calls
I20260621 05:10:51.849028 11637 log.cc:824] T dd732866e60a418a8cd9b605f30c77e3 P 6fa73262ba394487a5a54708f5de7075: Log is configured to *not* fsync() on all Append() calls
I20260621 05:10:51.849208 11634 raft_consensus.cc:348] T dd732866e60a418a8cd9b605f30c77e3 P eceee696a5ee4011b1264da66419eae9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eceee696a5ee4011b1264da66419eae9" member_type: VOTER last_known_addr { host: "127.3.131.130" port: 36509 } } peers { permanent_uuid: "017e3652ab544ab08232482357ea8d48" member_type: VOTER last_known_addr { host: "127.3.131.131" port: 46349 } } peers { permanent_uuid: "6fa73262ba394487a5a54708f5de7075" member_type: VOTER last_known_addr { host: "127.3.131.129" port: 38555 } }
I20260621 05:10:51.849421 11634 raft_consensus.cc:374] T dd732866e60a418a8cd9b605f30c77e3 P eceee696a5ee4011b1264da66419eae9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 05:10:51.849460 11634 raft_consensus.cc:729] T dd732866e60a418a8cd9b605f30c77e3 P eceee696a5ee4011b1264da66419eae9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eceee696a5ee4011b1264da66419eae9, State: Initialized, Role: FOLLOWER
I20260621 05:10:51.849617 11635 tablet_bootstrap.cc:492] T dd732866e60a418a8cd9b605f30c77e3 P 017e3652ab544ab08232482357ea8d48: No bootstrap required, opened a new log
I20260621 05:10:51.849617 11634 consensus_queue.cc:260] T dd732866e60a418a8cd9b605f30c77e3 P eceee696a5ee4011b1264da66419eae9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eceee696a5ee4011b1264da66419eae9" member_type: VOTER last_known_addr { host: "127.3.131.130" port: 36509 } } peers { permanent_uuid: "017e3652ab544ab08232482357ea8d48" member_type: VOTER last_known_addr { host: "127.3.131.131" port: 46349 } } peers { permanent_uuid: "6fa73262ba394487a5a54708f5de7075" member_type: VOTER last_known_addr { host: "127.3.131.129" port: 38555 } }
I20260621 05:10:51.849687 11635 ts_tablet_manager.cc:1397] T dd732866e60a418a8cd9b605f30c77e3 P 017e3652ab544ab08232482357ea8d48: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260621 05:10:51.849750 11637 tablet_bootstrap.cc:492] T dd732866e60a418a8cd9b605f30c77e3 P 6fa73262ba394487a5a54708f5de7075: No bootstrap required, opened a new log
I20260621 05:10:51.849824 11637 ts_tablet_manager.cc:1397] T dd732866e60a418a8cd9b605f30c77e3 P 6fa73262ba394487a5a54708f5de7075: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260621 05:10:51.849896 11634 ts_tablet_manager.cc:1428] T dd732866e60a418a8cd9b605f30c77e3 P eceee696a5ee4011b1264da66419eae9: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260621 05:10:51.850047 11466 heartbeater.cc:503] Master 127.3.131.190:33031 was elected leader, sending a full tablet report...
I20260621 05:10:51.851859 11637 raft_consensus.cc:348] T dd732866e60a418a8cd9b605f30c77e3 P 6fa73262ba394487a5a54708f5de7075 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eceee696a5ee4011b1264da66419eae9" member_type: VOTER last_known_addr { host: "127.3.131.130" port: 36509 } } peers { permanent_uuid: "017e3652ab544ab08232482357ea8d48" member_type: VOTER last_known_addr { host: "127.3.131.131" port: 46349 } } peers { permanent_uuid: "6fa73262ba394487a5a54708f5de7075" member_type: VOTER last_known_addr { host: "127.3.131.129" port: 38555 } }
I20260621 05:10:51.852000 11637 raft_consensus.cc:374] T dd732866e60a418a8cd9b605f30c77e3 P 6fa73262ba394487a5a54708f5de7075 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 05:10:51.852022 11637 raft_consensus.cc:729] T dd732866e60a418a8cd9b605f30c77e3 P 6fa73262ba394487a5a54708f5de7075 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6fa73262ba394487a5a54708f5de7075, State: Initialized, Role: FOLLOWER
I20260621 05:10:51.851903 11635 raft_consensus.cc:348] T dd732866e60a418a8cd9b605f30c77e3 P 017e3652ab544ab08232482357ea8d48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eceee696a5ee4011b1264da66419eae9" member_type: VOTER last_known_addr { host: "127.3.131.130" port: 36509 } } peers { permanent_uuid: "017e3652ab544ab08232482357ea8d48" member_type: VOTER last_known_addr { host: "127.3.131.131" port: 46349 } } peers { permanent_uuid: "6fa73262ba394487a5a54708f5de7075" member_type: VOTER last_known_addr { host: "127.3.131.129" port: 38555 } }
I20260621 05:10:51.852077 11635 raft_consensus.cc:374] T dd732866e60a418a8cd9b605f30c77e3 P 017e3652ab544ab08232482357ea8d48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 05:10:51.852111 11635 raft_consensus.cc:729] T dd732866e60a418a8cd9b605f30c77e3 P 017e3652ab544ab08232482357ea8d48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 017e3652ab544ab08232482357ea8d48, State: Initialized, Role: FOLLOWER
I20260621 05:10:51.852151 11637 consensus_queue.cc:260] T dd732866e60a418a8cd9b605f30c77e3 P 6fa73262ba394487a5a54708f5de7075 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eceee696a5ee4011b1264da66419eae9" member_type: VOTER last_known_addr { host: "127.3.131.130" port: 36509 } } peers { permanent_uuid: "017e3652ab544ab08232482357ea8d48" member_type: VOTER last_known_addr { host: "127.3.131.131" port: 46349 } } peers { permanent_uuid: "6fa73262ba394487a5a54708f5de7075" member_type: VOTER last_known_addr { host: "127.3.131.129" port: 38555 } }
I20260621 05:10:51.852231 11635 consensus_queue.cc:260] T dd732866e60a418a8cd9b605f30c77e3 P 017e3652ab544ab08232482357ea8d48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eceee696a5ee4011b1264da66419eae9" member_type: VOTER last_known_addr { host: "127.3.131.130" port: 36509 } } peers { permanent_uuid: "017e3652ab544ab08232482357ea8d48" member_type: VOTER last_known_addr { host: "127.3.131.131" port: 46349 } } peers { permanent_uuid: "6fa73262ba394487a5a54708f5de7075" member_type: VOTER last_known_addr { host: "127.3.131.129" port: 38555 } }
I20260621 05:10:51.852444 11637 ts_tablet_manager.cc:1428] T dd732866e60a418a8cd9b605f30c77e3 P 6fa73262ba394487a5a54708f5de7075: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260621 05:10:51.852449 11326 heartbeater.cc:503] Master 127.3.131.190:33031 was elected leader, sending a full tablet report...
I20260621 05:10:51.852499 11635 ts_tablet_manager.cc:1428] T dd732866e60a418a8cd9b605f30c77e3 P 017e3652ab544ab08232482357ea8d48: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260621 05:10:51.852504 11606 heartbeater.cc:503] Master 127.3.131.190:33031 was elected leader, sending a full tablet report...
W20260621 05:10:51.873910 11467 tablet.cc:2369] T dd732866e60a418a8cd9b605f30c77e3 P eceee696a5ee4011b1264da66419eae9: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260621 05:10:51.925228 11640 raft_consensus.cc:482] T dd732866e60a418a8cd9b605f30c77e3 P eceee696a5ee4011b1264da66419eae9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260621 05:10:51.925360 11640 raft_consensus.cc:504] T dd732866e60a418a8cd9b605f30c77e3 P eceee696a5ee4011b1264da66419eae9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eceee696a5ee4011b1264da66419eae9" member_type: VOTER last_known_addr { host: "127.3.131.130" port: 36509 } } peers { permanent_uuid: "017e3652ab544ab08232482357ea8d48" member_type: VOTER last_known_addr { host: "127.3.131.131" port: 46349 } } peers { permanent_uuid: "6fa73262ba394487a5a54708f5de7075" member_type: VOTER last_known_addr { host: "127.3.131.129" port: 38555 } }
I20260621 05:10:51.925676 11640 leader_election.cc:288] T dd732866e60a418a8cd9b605f30c77e3 P eceee696a5ee4011b1264da66419eae9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 017e3652ab544ab08232482357ea8d48 (127.3.131.131:46349), 6fa73262ba394487a5a54708f5de7075 (127.3.131.129:38555)
I20260621 05:10:51.928951 11632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:10:51.926112 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.131.130:45551 (local address 127.3.131.131:46349)
0621 05:10:51.926157 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0621 05:10:51.926159 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:10:51.926275 (+ 116us) server_negotiation.cc:407] Connection header received
0621 05:10:51.926357 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:10:51.926359 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:10:51.926369 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:10:51.926442 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 05:10:51.926606 (+ 164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:10:51.927089 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:10:51.928140 (+ 1051us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:10:51.928559 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:10:51.928658 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:10:51.928675 (+ 17us) server_negotiation.cc:1035] Waiting for connection context
0621 05:10:51.928839 (+ 164us) server_negotiation.cc:299] Negotiation successful
0621 05:10:51.928877 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260621 05:10:51.928952 11359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:10:51.926032 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.131.131:46349 (local address 127.3.131.130:45551)
0621 05:10:51.926141 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0621 05:10:51.926155 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0621 05:10:51.926283 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:10:51.926460 (+ 177us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:10:51.926461 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:10:51.926462 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 05:10:51.926537 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:10:51.926539 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:10:51.927217 (+ 678us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:10:51.927219 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:10:51.928042 (+ 823us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:10:51.928045 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:10:51.928657 (+ 612us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:10:51.928658 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:10:51.928755 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:10:51.928758 (+ 3us) client_negotiation.cc:769] Sending connection context
0621 05:10:51.928831 (+ 73us) client_negotiation.cc:240] Negotiation successful
0621 05:10:51.928874 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":86,"mutex_wait_us":50}
I20260621 05:10:51.929021 11631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:10:51.926210 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.131.130:59549 (local address 127.3.131.129:38555)
0621 05:10:51.926288 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0621 05:10:51.926290 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:10:51.926428 (+ 138us) server_negotiation.cc:407] Connection header received
0621 05:10:51.926529 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:10:51.926530 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:10:51.926540 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:10:51.926588 (+ 48us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 05:10:51.926761 (+ 173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:10:51.927244 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:10:51.928176 (+ 932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:10:51.928612 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:10:51.928703 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:10:51.928711 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0621 05:10:51.928907 (+ 196us) server_negotiation.cc:299] Negotiation successful
0621 05:10:51.928944 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260621 05:10:51.929075 11643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:10:51.926076 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.131.129:38555 (local address 127.3.131.130:59549)
0621 05:10:51.926352 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0621 05:10:51.926361 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0621 05:10:51.926442 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:10:51.926607 (+ 165us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:10:51.926608 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:10:51.926610 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 05:10:51.926693 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:10:51.926695 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:10:51.927351 (+ 656us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:10:51.927352 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:10:51.928086 (+ 734us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:10:51.928087 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:10:51.928704 (+ 617us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:10:51.928705 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:10:51.928782 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:10:51.928785 (+ 3us) client_negotiation.cc:769] Sending connection context
0621 05:10:51.928898 (+ 113us) client_negotiation.cc:240] Negotiation successful
0621 05:10:51.928943 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":254,"mutex_wait_us":84,"thread_start_us":121,"threads_started":1}
I20260621 05:10:51.929454 11277 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd732866e60a418a8cd9b605f30c77e3" candidate_uuid: "eceee696a5ee4011b1264da66419eae9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6fa73262ba394487a5a54708f5de7075" is_pre_election: true
I20260621 05:10:51.929658 11277 raft_consensus.cc:2454] T dd732866e60a418a8cd9b605f30c77e3 P 6fa73262ba394487a5a54708f5de7075 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eceee696a5ee4011b1264da66419eae9 in term 0.
I20260621 05:10:51.929931 11351 leader_election.cc:302] T dd732866e60a418a8cd9b605f30c77e3 P eceee696a5ee4011b1264da66419eae9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6fa73262ba394487a5a54708f5de7075, eceee696a5ee4011b1264da66419eae9; no voters:
I20260621 05:10:51.929855 11560 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd732866e60a418a8cd9b605f30c77e3" candidate_uuid: "eceee696a5ee4011b1264da66419eae9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "017e3652ab544ab08232482357ea8d48" is_pre_election: true
I20260621 05:10:51.930049 11560 raft_consensus.cc:2454] T dd732866e60a418a8cd9b605f30c77e3 P 017e3652ab544ab08232482357ea8d48 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eceee696a5ee4011b1264da66419eae9 in term 0.
I20260621 05:10:51.930074 11640 raft_consensus.cc:2781] T dd732866e60a418a8cd9b605f30c77e3 P eceee696a5ee4011b1264da66419eae9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260621 05:10:51.930132 11640 raft_consensus.cc:482] T dd732866e60a418a8cd9b605f30c77e3 P eceee696a5ee4011b1264da66419eae9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260621 05:10:51.930167 11640 raft_consensus.cc:3037] T dd732866e60a418a8cd9b605f30c77e3 P eceee696a5ee4011b1264da66419eae9 [term 0 FOLLOWER]: Advancing to term 1
I20260621 05:10:51.931445 11640 raft_consensus.cc:504] T dd732866e60a418a8cd9b605f30c77e3 P eceee696a5ee4011b1264da66419eae9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eceee696a5ee4011b1264da66419eae9" member_type: VOTER last_known_addr { host: "127.3.131.130" port: 36509 } } peers { permanent_uuid: "017e3652ab544ab08232482357ea8d48" member_type: VOTER last_known_addr { host: "127.3.131.131" port: 46349 } } peers { permanent_uuid: "6fa73262ba394487a5a54708f5de7075" member_type: VOTER last_known_addr { host: "127.3.131.129" port: 38555 } }
I20260621 05:10:51.931595 11640 leader_election.cc:288] T dd732866e60a418a8cd9b605f30c77e3 P eceee696a5ee4011b1264da66419eae9 [CANDIDATE]: Term 1 election: Requested vote from peers 017e3652ab544ab08232482357ea8d48 (127.3.131.131:46349), 6fa73262ba394487a5a54708f5de7075 (127.3.131.129:38555)
I20260621 05:10:51.931748 11560 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd732866e60a418a8cd9b605f30c77e3" candidate_uuid: "eceee696a5ee4011b1264da66419eae9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "017e3652ab544ab08232482357ea8d48"
I20260621 05:10:51.931829 11560 raft_consensus.cc:3037] T dd732866e60a418a8cd9b605f30c77e3 P 017e3652ab544ab08232482357ea8d48 [term 0 FOLLOWER]: Advancing to term 1
I20260621 05:10:51.931773 11277 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd732866e60a418a8cd9b605f30c77e3" candidate_uuid: "eceee696a5ee4011b1264da66419eae9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6fa73262ba394487a5a54708f5de7075"
I20260621 05:10:51.931886 11277 raft_consensus.cc:3037] T dd732866e60a418a8cd9b605f30c77e3 P 6fa73262ba394487a5a54708f5de7075 [term 0 FOLLOWER]: Advancing to term 1
I20260621 05:10:51.932559 11560 raft_consensus.cc:2454] T dd732866e60a418a8cd9b605f30c77e3 P 017e3652ab544ab08232482357ea8d48 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eceee696a5ee4011b1264da66419eae9 in term 1.
I20260621 05:10:51.932740 11352 leader_election.cc:302] T dd732866e60a418a8cd9b605f30c77e3 P eceee696a5ee4011b1264da66419eae9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 017e3652ab544ab08232482357ea8d48, eceee696a5ee4011b1264da66419eae9; no voters:
I20260621 05:10:51.932847 11277 raft_consensus.cc:2454] T dd732866e60a418a8cd9b605f30c77e3 P 6fa73262ba394487a5a54708f5de7075 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eceee696a5ee4011b1264da66419eae9 in term 1.
I20260621 05:10:51.932869 11640 raft_consensus.cc:2781] T dd732866e60a418a8cd9b605f30c77e3 P eceee696a5ee4011b1264da66419eae9 [term 1 FOLLOWER]: Leader election won for term 1
I20260621 05:10:51.933017 11640 raft_consensus.cc:686] T dd732866e60a418a8cd9b605f30c77e3 P eceee696a5ee4011b1264da66419eae9 [term 1 LEADER]: Becoming Leader. State: Replica: eceee696a5ee4011b1264da66419eae9, State: Running, Role: LEADER
I20260621 05:10:51.933117 11640 consensus_queue.cc:237] T dd732866e60a418a8cd9b605f30c77e3 P eceee696a5ee4011b1264da66419eae9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eceee696a5ee4011b1264da66419eae9" member_type: VOTER last_known_addr { host: "127.3.131.130" port: 36509 } } peers { permanent_uuid: "017e3652ab544ab08232482357ea8d48" member_type: VOTER last_known_addr { host: "127.3.131.131" port: 46349 } } peers { permanent_uuid: "6fa73262ba394487a5a54708f5de7075" member_type: VOTER last_known_addr { host: "127.3.131.129" port: 38555 } }
I20260621 05:10:51.933915 11122 catalog_manager.cc:5495] T dd732866e60a418a8cd9b605f30c77e3 P eceee696a5ee4011b1264da66419eae9 reported cstate change: term changed from 0 to 1, leader changed from <none> to eceee696a5ee4011b1264da66419eae9 (127.3.131.130). New cstate: current_term: 1 leader_uuid: "eceee696a5ee4011b1264da66419eae9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eceee696a5ee4011b1264da66419eae9" member_type: VOTER last_known_addr { host: "127.3.131.130" port: 36509 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "017e3652ab544ab08232482357ea8d48" member_type: VOTER last_known_addr { host: "127.3.131.131" port: 46349 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6fa73262ba394487a5a54708f5de7075" member_type: VOTER last_known_addr { host: "127.3.131.129" port: 38555 } health_report { overall_health: UNKNOWN } } }
W20260621 05:10:51.955227 11327 tablet.cc:2369] T dd732866e60a418a8cd9b605f30c77e3 P 6fa73262ba394487a5a54708f5de7075: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260621 05:10:51.961758 11633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:10:51.958848 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35270 (local address 127.3.131.130:36509)
0621 05:10:51.958919 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0621 05:10:51.958922 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0621 05:10:51.958931 (+ 9us) server_negotiation.cc:407] Connection header received
0621 05:10:51.958982 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:10:51.958983 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:10:51.959001 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:10:51.959067 (+ 66us) server_negotiation.cc:226] Negotiated authn=TOKEN
0621 05:10:51.959297 (+ 230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:10:51.959844 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:10:51.960748 (+ 904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:10:51.961021 (+ 273us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:10:51.961180 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:10:51.961383 (+ 203us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0621 05:10:51.961493 (+ 110us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0621 05:10:51.961585 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0621 05:10:51.961645 (+ 60us) server_negotiation.cc:299] Negotiation successful
0621 05:10:51.961668 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260621 05:10:51.963946 11560 raft_consensus.cc:1261] T dd732866e60a418a8cd9b605f30c77e3 P 017e3652ab544ab08232482357ea8d48 [term 1 FOLLOWER]: Refusing update from remote peer eceee696a5ee4011b1264da66419eae9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 05:10:51.963946 11277 raft_consensus.cc:1261] T dd732866e60a418a8cd9b605f30c77e3 P 6fa73262ba394487a5a54708f5de7075 [term 1 FOLLOWER]: Refusing update from remote peer eceee696a5ee4011b1264da66419eae9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 05:10:51.964403 11644 consensus_queue.cc:1035] T dd732866e60a418a8cd9b605f30c77e3 P eceee696a5ee4011b1264da66419eae9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "017e3652ab544ab08232482357ea8d48" member_type: VOTER last_known_addr { host: "127.3.131.131" port: 46349 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 05:10:51.964578 11640 consensus_queue.cc:1035] T dd732866e60a418a8cd9b605f30c77e3 P eceee696a5ee4011b1264da66419eae9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6fa73262ba394487a5a54708f5de7075" member_type: VOTER last_known_addr { host: "127.3.131.129" port: 38555 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 05:10:51.967619 11649 mvcc.cc:205] Tried to move back new op lower bound from 7299148398441013248 to 7299148398318448640. Current Snapshot: MvccSnapshot[applied={T|T < 7299148398441013248 or (T in {7299148398441013248})}]
I20260621 05:10:51.969081 11122 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:57290:
table { table_name: "test-table" } modify_external_catalogs: true
I20260621 05:10:51.969185 11122 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:57290:
table { table_name: "test-table" } modify_external_catalogs: true
I20260621 05:10:51.969225 11648 mvcc.cc:205] Tried to move back new op lower bound from 7299148398441013248 to 7299148398318448640. Current Snapshot: MvccSnapshot[applied={T|T < 7299148398441013248}]
I20260621 05:10:51.969959 11122 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 65139302364d411cb10a6e712d783f40: Sending DeleteTablet for 3 replicas of tablet dd732866e60a418a8cd9b605f30c77e3
I20260621 05:10:51.970211 11260 tablet_service.cc:1514] Processing DeleteTablet for tablet dd732866e60a418a8cd9b605f30c77e3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 05:10:51 UTC) from {username='oryx', principal='oryx/127.3.131.190@KRBTEST.COM'} at 127.0.0.1:41098
I20260621 05:10:51.970383 11400 tablet_service.cc:1514] Processing DeleteTablet for tablet dd732866e60a418a8cd9b605f30c77e3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 05:10:51 UTC) from {username='oryx', principal='oryx/127.3.131.190@KRBTEST.COM'} at 127.0.0.1:35264
I20260621 05:10:51.970480 11122 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:57290:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260621 05:10:51.970592 11656 tablet_replica.cc:330] T dd732866e60a418a8cd9b605f30c77e3 P 6fa73262ba394487a5a54708f5de7075: stopping tablet replica
W20260621 05:10:51.970615 11122 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260621 05:10:51.970610 11540 tablet_service.cc:1514] Processing DeleteTablet for tablet dd732866e60a418a8cd9b605f30c77e3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 05:10:51 UTC) from {username='oryx', principal='oryx/127.3.131.190@KRBTEST.COM'} at 127.0.0.1:38122
I20260621 05:10:51.970682 11656 raft_consensus.cc:2229] T dd732866e60a418a8cd9b605f30c77e3 P 6fa73262ba394487a5a54708f5de7075 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260621 05:10:51.970779 11656 raft_consensus.cc:2258] T dd732866e60a418a8cd9b605f30c77e3 P 6fa73262ba394487a5a54708f5de7075 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 05:10:51.970937 11657 tablet_replica.cc:330] T dd732866e60a418a8cd9b605f30c77e3 P eceee696a5ee4011b1264da66419eae9: stopping tablet replica
I20260621 05:10:51.971015 11657 raft_consensus.cc:2229] T dd732866e60a418a8cd9b605f30c77e3 P eceee696a5ee4011b1264da66419eae9 [term 1 LEADER]: Raft consensus shutting down.
I20260621 05:10:51.971005 11539 tablet_service.cc:1514] Processing DeleteTablet for tablet dd732866e60a418a8cd9b605f30c77e3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 05:10:51 UTC) from {username='oryx', principal='oryx/127.3.131.190@KRBTEST.COM'} at 127.0.0.1:38122
I20260621 05:10:51.971105 11657 raft_consensus.cc:2258] T dd732866e60a418a8cd9b605f30c77e3 P eceee696a5ee4011b1264da66419eae9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 05:10:51.971451 11658 tablet_replica.cc:330] T dd732866e60a418a8cd9b605f30c77e3 P 017e3652ab544ab08232482357ea8d48: stopping tablet replica
I20260621 05:10:51.971558 11658 raft_consensus.cc:2229] T dd732866e60a418a8cd9b605f30c77e3 P 017e3652ab544ab08232482357ea8d48 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260621 05:10:51.971659 11658 raft_consensus.cc:2258] T dd732866e60a418a8cd9b605f30c77e3 P 017e3652ab544ab08232482357ea8d48 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 05:10:51.971756 11656 ts_tablet_manager.cc:1905] T dd732866e60a418a8cd9b605f30c77e3 P 6fa73262ba394487a5a54708f5de7075: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 05:10:51.972254 11657 ts_tablet_manager.cc:1905] T dd732866e60a418a8cd9b605f30c77e3 P eceee696a5ee4011b1264da66419eae9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 05:10:51.972766 11260 tablet_service.cc:1467] Processing CreateTablet for tablet 3860238ba18542c182c642d4ae7d456a (DEFAULT_TABLE table=test-table [id=e34454efe12843a6a7f72345f02c395e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 05:10:51.972926 11400 tablet_service.cc:1467] Processing CreateTablet for tablet 3860238ba18542c182c642d4ae7d456a (DEFAULT_TABLE table=test-table [id=e34454efe12843a6a7f72345f02c395e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 05:10:51.972990 11260 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3860238ba18542c182c642d4ae7d456a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 05:10:51.973028 11400 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3860238ba18542c182c642d4ae7d456a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 05:10:51.973946 11657 ts_tablet_manager.cc:1918] T dd732866e60a418a8cd9b605f30c77e3 P eceee696a5ee4011b1264da66419eae9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260621 05:10:51.974009 11657 log.cc:1196] T dd732866e60a418a8cd9b605f30c77e3 P eceee696a5ee4011b1264da66419eae9: Deleting WAL directory at /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782018616648578-3598-0/minicluster-data/ts-1/wal/wals/dd732866e60a418a8cd9b605f30c77e3
I20260621 05:10:51.974261 11657 ts_tablet_manager.cc:1939] T dd732866e60a418a8cd9b605f30c77e3 P eceee696a5ee4011b1264da66419eae9: Deleting consensus metadata
I20260621 05:10:51.974555 11634 tablet_bootstrap.cc:492] T 3860238ba18542c182c642d4ae7d456a P eceee696a5ee4011b1264da66419eae9: Bootstrap starting.
I20260621 05:10:51.974623 11539 tablet_service.cc:1467] Processing CreateTablet for tablet 3860238ba18542c182c642d4ae7d456a (DEFAULT_TABLE table=test-table [id=e34454efe12843a6a7f72345f02c395e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 05:10:51.974730 11539 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3860238ba18542c182c642d4ae7d456a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 05:10:51.974911 11634 tablet_bootstrap.cc:654] T 3860238ba18542c182c642d4ae7d456a P eceee696a5ee4011b1264da66419eae9: Neither blocks nor log segments found. Creating new log.
I20260621 05:10:51.975244 11637 tablet_bootstrap.cc:492] T 3860238ba18542c182c642d4ae7d456a P 6fa73262ba394487a5a54708f5de7075: Bootstrap starting.
I20260621 05:10:51.975507 11634 tablet_bootstrap.cc:492] T 3860238ba18542c182c642d4ae7d456a P eceee696a5ee4011b1264da66419eae9: No bootstrap required, opened a new log
I20260621 05:10:51.975564 11634 ts_tablet_manager.cc:1397] T 3860238ba18542c182c642d4ae7d456a P eceee696a5ee4011b1264da66419eae9: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260621 05:10:51.975669 11637 tablet_bootstrap.cc:654] T 3860238ba18542c182c642d4ae7d456a P 6fa73262ba394487a5a54708f5de7075: Neither blocks nor log segments found. Creating new log.
I20260621 05:10:51.975729 11634 raft_consensus.cc:348] T 3860238ba18542c182c642d4ae7d456a P eceee696a5ee4011b1264da66419eae9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fa73262ba394487a5a54708f5de7075" member_type: VOTER last_known_addr { host: "127.3.131.129" port: 38555 } } peers { permanent_uuid: "eceee696a5ee4011b1264da66419eae9" member_type: VOTER last_known_addr { host: "127.3.131.130" port: 36509 } } peers { permanent_uuid: "017e3652ab544ab08232482357ea8d48" member_type: VOTER last_known_addr { host: "127.3.131.131" port: 46349 } }
I20260621 05:10:51.975808 11634 raft_consensus.cc:374] T 3860238ba18542c182c642d4ae7d456a P eceee696a5ee4011b1264da66419eae9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 05:10:51.975835 11634 raft_consensus.cc:729] T 3860238ba18542c182c642d4ae7d456a P eceee696a5ee4011b1264da66419eae9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eceee696a5ee4011b1264da66419eae9, State: Initialized, Role: FOLLOWER
I20260621 05:10:51.975883 11634 consensus_queue.cc:260] T 3860238ba18542c182c642d4ae7d456a P eceee696a5ee4011b1264da66419eae9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fa73262ba394487a5a54708f5de7075" member_type: VOTER last_known_addr { host: "127.3.131.129" port: 38555 } } peers { permanent_uuid: "eceee696a5ee4011b1264da66419eae9" member_type: VOTER last_known_addr { host: "127.3.131.130" port: 36509 } } peers { permanent_uuid: "017e3652ab544ab08232482357ea8d48" member_type: VOTER last_known_addr { host: "127.3.131.131" port: 46349 } }
I20260621 05:10:51.975984 11634 ts_tablet_manager.cc:1428] T 3860238ba18542c182c642d4ae7d456a P eceee696a5ee4011b1264da66419eae9: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260621 05:10:51.976305 11109 catalog_manager.cc:4841] TS eceee696a5ee4011b1264da66419eae9 (127.3.131.130:36509): tablet dd732866e60a418a8cd9b605f30c77e3 (table test-table [id=3a7e99a0a23748148205ed74756e5043]) successfully deleted
I20260621 05:10:51.976500 11656 ts_tablet_manager.cc:1918] T dd732866e60a418a8cd9b605f30c77e3 P 6fa73262ba394487a5a54708f5de7075: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260621 05:10:51.976629 11656 log.cc:1196] T dd732866e60a418a8cd9b605f30c77e3 P 6fa73262ba394487a5a54708f5de7075: Deleting WAL directory at /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782018616648578-3598-0/minicluster-data/ts-0/wal/wals/dd732866e60a418a8cd9b605f30c77e3
I20260621 05:10:51.977010 11656 ts_tablet_manager.cc:1939] T dd732866e60a418a8cd9b605f30c77e3 P 6fa73262ba394487a5a54708f5de7075: Deleting consensus metadata
I20260621 05:10:51.977440 11107 catalog_manager.cc:4841] TS 6fa73262ba394487a5a54708f5de7075 (127.3.131.129:38555): tablet dd732866e60a418a8cd9b605f30c77e3 (table test-table [id=3a7e99a0a23748148205ed74756e5043]) successfully deleted
I20260621 05:10:51.977563 11635 tablet_bootstrap.cc:492] T 3860238ba18542c182c642d4ae7d456a P 017e3652ab544ab08232482357ea8d48: Bootstrap starting.
I20260621 05:10:51.977994 11635 tablet_bootstrap.cc:654] T 3860238ba18542c182c642d4ae7d456a P 017e3652ab544ab08232482357ea8d48: Neither blocks nor log segments found. Creating new log.
I20260621 05:10:51.978130 11658 ts_tablet_manager.cc:1905] T dd732866e60a418a8cd9b605f30c77e3 P 017e3652ab544ab08232482357ea8d48: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 05:10:51.978596 11635 tablet_bootstrap.cc:492] T 3860238ba18542c182c642d4ae7d456a P 017e3652ab544ab08232482357ea8d48: No bootstrap required, opened a new log
I20260621 05:10:51.978665 11635 ts_tablet_manager.cc:1397] T 3860238ba18542c182c642d4ae7d456a P 017e3652ab544ab08232482357ea8d48: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260621 05:10:51.978827 11635 raft_consensus.cc:348] T 3860238ba18542c182c642d4ae7d456a P 017e3652ab544ab08232482357ea8d48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fa73262ba394487a5a54708f5de7075" member_type: VOTER last_known_addr { host: "127.3.131.129" port: 38555 } } peers { permanent_uuid: "eceee696a5ee4011b1264da66419eae9" member_type: VOTER last_known_addr { host: "127.3.131.130" port: 36509 } } peers { permanent_uuid: "017e3652ab544ab08232482357ea8d48" member_type: VOTER last_known_addr { host: "127.3.131.131" port: 46349 } }
I20260621 05:10:51.978912 11635 raft_consensus.cc:374] T 3860238ba18542c182c642d4ae7d456a P 017e3652ab544ab08232482357ea8d48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 05:10:51.978941 11635 raft_consensus.cc:729] T 3860238ba18542c182c642d4ae7d456a P 017e3652ab544ab08232482357ea8d48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 017e3652ab544ab08232482357ea8d48, State: Initialized, Role: FOLLOWER
I20260621 05:10:51.978986 11635 consensus_queue.cc:260] T 3860238ba18542c182c642d4ae7d456a P 017e3652ab544ab08232482357ea8d48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fa73262ba394487a5a54708f5de7075" member_type: VOTER last_known_addr { host: "127.3.131.129" port: 38555 } } peers { permanent_uuid: "eceee696a5ee4011b1264da66419eae9" member_type: VOTER last_known_addr { host: "127.3.131.130" port: 36509 } } peers { permanent_uuid: "017e3652ab544ab08232482357ea8d48" member_type: VOTER last_known_addr { host: "127.3.131.131" port: 46349 } }
I20260621 05:10:51.979082 11635 ts_tablet_manager.cc:1428] T 3860238ba18542c182c642d4ae7d456a P 017e3652ab544ab08232482357ea8d48: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260621 05:10:51.979883 11658 ts_tablet_manager.cc:1918] T dd732866e60a418a8cd9b605f30c77e3 P 017e3652ab544ab08232482357ea8d48: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260621 05:10:51.979964 11658 log.cc:1196] T dd732866e60a418a8cd9b605f30c77e3 P 017e3652ab544ab08232482357ea8d48: Deleting WAL directory at /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782018616648578-3598-0/minicluster-data/ts-2/wal/wals/dd732866e60a418a8cd9b605f30c77e3
I20260621 05:10:51.980356 11658 ts_tablet_manager.cc:1939] T dd732866e60a418a8cd9b605f30c77e3 P 017e3652ab544ab08232482357ea8d48: Deleting consensus metadata
I20260621 05:10:51.981245 11108 catalog_manager.cc:4841] TS 017e3652ab544ab08232482357ea8d48 (127.3.131.131:46349): tablet dd732866e60a418a8cd9b605f30c77e3 (table test-table [id=3a7e99a0a23748148205ed74756e5043]) successfully deleted
W20260621 05:10:51.981431 11108 catalog_manager.cc:4808] TS 017e3652ab544ab08232482357ea8d48 (127.3.131.131:46349): delete failed for tablet dd732866e60a418a8cd9b605f30c77e3 because the tablet was not found. No further retry: Not found: Tablet not found: dd732866e60a418a8cd9b605f30c77e3
I20260621 05:10:51.983700 11637 tablet_bootstrap.cc:492] T 3860238ba18542c182c642d4ae7d456a P 6fa73262ba394487a5a54708f5de7075: No bootstrap required, opened a new log
I20260621 05:10:51.983774 11637 ts_tablet_manager.cc:1397] T 3860238ba18542c182c642d4ae7d456a P 6fa73262ba394487a5a54708f5de7075: Time spent bootstrapping tablet: real 0.009s user 0.001s sys 0.000s
I20260621 05:10:51.984197 11637 raft_consensus.cc:348] T 3860238ba18542c182c642d4ae7d456a P 6fa73262ba394487a5a54708f5de7075 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fa73262ba394487a5a54708f5de7075" member_type: VOTER last_known_addr { host: "127.3.131.129" port: 38555 } } peers { permanent_uuid: "eceee696a5ee4011b1264da66419eae9" member_type: VOTER last_known_addr { host: "127.3.131.130" port: 36509 } } peers { permanent_uuid: "017e3652ab544ab08232482357ea8d48" member_type: VOTER last_known_addr { host: "127.3.131.131" port: 46349 } }
I20260621 05:10:51.984305 11637 raft_consensus.cc:374] T 3860238ba18542c182c642d4ae7d456a P 6fa73262ba394487a5a54708f5de7075 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 05:10:51.984329 11637 raft_consensus.cc:729] T 3860238ba18542c182c642d4ae7d456a P 6fa73262ba394487a5a54708f5de7075 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6fa73262ba394487a5a54708f5de7075, State: Initialized, Role: FOLLOWER
I20260621 05:10:51.984385 11637 consensus_queue.cc:260] T 3860238ba18542c182c642d4ae7d456a P 6fa73262ba394487a5a54708f5de7075 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fa73262ba394487a5a54708f5de7075" member_type: VOTER last_known_addr { host: "127.3.131.129" port: 38555 } } peers { permanent_uuid: "eceee696a5ee4011b1264da66419eae9" member_type: VOTER last_known_addr { host: "127.3.131.130" port: 36509 } } peers { permanent_uuid: "017e3652ab544ab08232482357ea8d48" member_type: VOTER last_known_addr { host: "127.3.131.131" port: 46349 } }
I20260621 05:10:51.984503 11637 ts_tablet_manager.cc:1428] T 3860238ba18542c182c642d4ae7d456a P 6fa73262ba394487a5a54708f5de7075: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
W20260621 05:10:52.032881 11607 tablet.cc:2369] T 3860238ba18542c182c642d4ae7d456a P 017e3652ab544ab08232482357ea8d48: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260621 05:10:52.167743 11644 raft_consensus.cc:482] T 3860238ba18542c182c642d4ae7d456a P eceee696a5ee4011b1264da66419eae9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260621 05:10:52.167860 11644 raft_consensus.cc:504] T 3860238ba18542c182c642d4ae7d456a P eceee696a5ee4011b1264da66419eae9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fa73262ba394487a5a54708f5de7075" member_type: VOTER last_known_addr { host: "127.3.131.129" port: 38555 } } peers { permanent_uuid: "eceee696a5ee4011b1264da66419eae9" member_type: VOTER last_known_addr { host: "127.3.131.130" port: 36509 } } peers { permanent_uuid: "017e3652ab544ab08232482357ea8d48" member_type: VOTER last_known_addr { host: "127.3.131.131" port: 46349 } }
I20260621 05:10:52.168012 11644 leader_election.cc:288] T 3860238ba18542c182c642d4ae7d456a P eceee696a5ee4011b1264da66419eae9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6fa73262ba394487a5a54708f5de7075 (127.3.131.129:38555), 017e3652ab544ab08232482357ea8d48 (127.3.131.131:46349)
I20260621 05:10:52.168322 11277 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3860238ba18542c182c642d4ae7d456a" candidate_uuid: "eceee696a5ee4011b1264da66419eae9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6fa73262ba394487a5a54708f5de7075" is_pre_election: true
I20260621 05:10:52.168336 11560 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3860238ba18542c182c642d4ae7d456a" candidate_uuid: "eceee696a5ee4011b1264da66419eae9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "017e3652ab544ab08232482357ea8d48" is_pre_election: true
I20260621 05:10:52.168437 11560 raft_consensus.cc:2454] T 3860238ba18542c182c642d4ae7d456a P 017e3652ab544ab08232482357ea8d48 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eceee696a5ee4011b1264da66419eae9 in term 0.
I20260621 05:10:52.168438 11277 raft_consensus.cc:2454] T 3860238ba18542c182c642d4ae7d456a P 6fa73262ba394487a5a54708f5de7075 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eceee696a5ee4011b1264da66419eae9 in term 0.
I20260621 05:10:52.168627 11352 leader_election.cc:302] T 3860238ba18542c182c642d4ae7d456a P eceee696a5ee4011b1264da66419eae9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 017e3652ab544ab08232482357ea8d48, eceee696a5ee4011b1264da66419eae9; no voters:
I20260621 05:10:52.168754 11644 raft_consensus.cc:2781] T 3860238ba18542c182c642d4ae7d456a P eceee696a5ee4011b1264da66419eae9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260621 05:10:52.168828 11644 raft_consensus.cc:482] T 3860238ba18542c182c642d4ae7d456a P eceee696a5ee4011b1264da66419eae9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260621 05:10:52.168859 11644 raft_consensus.cc:3037] T 3860238ba18542c182c642d4ae7d456a P eceee696a5ee4011b1264da66419eae9 [term 0 FOLLOWER]: Advancing to term 1
I20260621 05:10:52.169615 11644 raft_consensus.cc:504] T 3860238ba18542c182c642d4ae7d456a P eceee696a5ee4011b1264da66419eae9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fa73262ba394487a5a54708f5de7075" member_type: VOTER last_known_addr { host: "127.3.131.129" port: 38555 } } peers { permanent_uuid: "eceee696a5ee4011b1264da66419eae9" member_type: VOTER last_known_addr { host: "127.3.131.130" port: 36509 } } peers { permanent_uuid: "017e3652ab544ab08232482357ea8d48" member_type: VOTER last_known_addr { host: "127.3.131.131" port: 46349 } }
I20260621 05:10:52.169780 11644 leader_election.cc:288] T 3860238ba18542c182c642d4ae7d456a P eceee696a5ee4011b1264da66419eae9 [CANDIDATE]: Term 1 election: Requested vote from peers 6fa73262ba394487a5a54708f5de7075 (127.3.131.129:38555), 017e3652ab544ab08232482357ea8d48 (127.3.131.131:46349)
I20260621 05:10:52.169953 11277 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3860238ba18542c182c642d4ae7d456a" candidate_uuid: "eceee696a5ee4011b1264da66419eae9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6fa73262ba394487a5a54708f5de7075"
I20260621 05:10:52.169960 11560 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3860238ba18542c182c642d4ae7d456a" candidate_uuid: "eceee696a5ee4011b1264da66419eae9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "017e3652ab544ab08232482357ea8d48"
I20260621 05:10:52.170054 11560 raft_consensus.cc:3037] T 3860238ba18542c182c642d4ae7d456a P 017e3652ab544ab08232482357ea8d48 [term 0 FOLLOWER]: Advancing to term 1
I20260621 05:10:52.170058 11277 raft_consensus.cc:3037] T 3860238ba18542c182c642d4ae7d456a P 6fa73262ba394487a5a54708f5de7075 [term 0 FOLLOWER]: Advancing to term 1
I20260621 05:10:52.170734 11277 raft_consensus.cc:2454] T 3860238ba18542c182c642d4ae7d456a P 6fa73262ba394487a5a54708f5de7075 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eceee696a5ee4011b1264da66419eae9 in term 1.
I20260621 05:10:52.170737 11560 raft_consensus.cc:2454] T 3860238ba18542c182c642d4ae7d456a P 017e3652ab544ab08232482357ea8d48 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eceee696a5ee4011b1264da66419eae9 in term 1.
I20260621 05:10:52.170948 11352 leader_election.cc:302] T 3860238ba18542c182c642d4ae7d456a P eceee696a5ee4011b1264da66419eae9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 017e3652ab544ab08232482357ea8d48, eceee696a5ee4011b1264da66419eae9; no voters:
I20260621 05:10:52.171074 11644 raft_consensus.cc:2781] T 3860238ba18542c182c642d4ae7d456a P eceee696a5ee4011b1264da66419eae9 [term 1 FOLLOWER]: Leader election won for term 1
I20260621 05:10:52.171137 11644 raft_consensus.cc:686] T 3860238ba18542c182c642d4ae7d456a P eceee696a5ee4011b1264da66419eae9 [term 1 LEADER]: Becoming Leader. State: Replica: eceee696a5ee4011b1264da66419eae9, State: Running, Role: LEADER
I20260621 05:10:52.171228 11644 consensus_queue.cc:237] T 3860238ba18542c182c642d4ae7d456a P eceee696a5ee4011b1264da66419eae9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fa73262ba394487a5a54708f5de7075" member_type: VOTER last_known_addr { host: "127.3.131.129" port: 38555 } } peers { permanent_uuid: "eceee696a5ee4011b1264da66419eae9" member_type: VOTER last_known_addr { host: "127.3.131.130" port: 36509 } } peers { permanent_uuid: "017e3652ab544ab08232482357ea8d48" member_type: VOTER last_known_addr { host: "127.3.131.131" port: 46349 } }
I20260621 05:10:52.171986 11123 catalog_manager.cc:5495] T 3860238ba18542c182c642d4ae7d456a P eceee696a5ee4011b1264da66419eae9 reported cstate change: term changed from 0 to 1, leader changed from <none> to eceee696a5ee4011b1264da66419eae9 (127.3.131.130). New cstate: current_term: 1 leader_uuid: "eceee696a5ee4011b1264da66419eae9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fa73262ba394487a5a54708f5de7075" member_type: VOTER last_known_addr { host: "127.3.131.129" port: 38555 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "eceee696a5ee4011b1264da66419eae9" member_type: VOTER last_known_addr { host: "127.3.131.130" port: 36509 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "017e3652ab544ab08232482357ea8d48" member_type: VOTER last_known_addr { host: "127.3.131.131" port: 46349 } health_report { overall_health: UNKNOWN } } }
Jun 21 05:10:52 dist-test-slave-vrq1 krb5kdc[11078](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782018651, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.131.190@KRBTEST.COM for oryx/127.3.131.190@KRBTEST.COM
I20260621 05:10:52.240890 11668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:10:52.232041 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.131.190:33031 (local address 127.0.0.1:57298)
0621 05:10:52.232228 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0621 05:10:52.232241 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0621 05:10:52.232314 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:10:52.232584 (+ 270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:10:52.232585 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:10:52.233134 (+ 549us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:10:52.233280 (+ 146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:10:52.233290 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:10:52.234179 (+ 889us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:10:52.234180 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:10:52.234747 (+ 567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:10:52.234749 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:10:52.235151 (+ 402us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:10:52.235152 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:10:52.235289 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:10:52.235980 (+ 691us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 05:10:52.235986 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:10:52.237770 (+ 1784us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 05:10:52.239699 (+ 1929us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:10:52.239707 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:10:52.239709 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:10:52.239935 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:10:52.240170 (+ 235us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:10:52.240171 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:10:52.240172 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:10:52.240240 (+ 68us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:10:52.240490 (+ 250us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 05:10:52.240527 (+ 37us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 05:10:52.240589 (+ 62us) client_negotiation.cc:769] Sending connection context
0621 05:10:52.240656 (+ 67us) client_negotiation.cc:240] Negotiation successful
0621 05:10:52.240768 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":92,"threads_started":1}
I20260621 05:10:52.240968 11174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:10:52.232097 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57298 (local address 127.3.131.190:33031)
0621 05:10:52.232149 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0621 05:10:52.232151 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:10:52.232357 (+ 206us) server_negotiation.cc:407] Connection header received
0621 05:10:52.232433 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:10:52.232434 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:10:52.232449 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:10:52.232531 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:10:52.233435 (+ 904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:10:52.234039 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:10:52.234861 (+ 822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:10:52.235052 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:10:52.235160 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:10:52.237883 (+ 2723us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:10:52.237885 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:10:52.237885 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:10:52.237893 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:10:52.239598 (+ 1705us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:10:52.240022 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:10:52.240023 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:10:52.240023 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:10:52.240078 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:10:52.240308 (+ 230us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:10:52.240308 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:10:52.240309 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:10:52.240423 (+ 114us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:10:52.240474 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0621 05:10:52.240719 (+ 245us) server_negotiation.cc:299] Negotiation successful
0621 05:10:52.240822 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":23,"server-negotiator.queue_time_us":26}
I20260621 05:10:52.243254 11123 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.3.131.190@KRBTEST.COM'} at 127.0.0.1:57298:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260621 05:10:52.243458 11123 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260621 05:10:52.244871 11400 tablet_service.cc:1467] Processing CreateTablet for tablet 9b19d4d48fbe409e82e2b1baf07079dd (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=589fd182c4b74547a7deea56f0f81844]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260621 05:10:52.244875 11260 tablet_service.cc:1467] Processing CreateTablet for tablet 9b19d4d48fbe409e82e2b1baf07079dd (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=589fd182c4b74547a7deea56f0f81844]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260621 05:10:52.244895 11539 tablet_service.cc:1467] Processing CreateTablet for tablet 9b19d4d48fbe409e82e2b1baf07079dd (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=589fd182c4b74547a7deea56f0f81844]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260621 05:10:52.245035 11539 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9b19d4d48fbe409e82e2b1baf07079dd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 05:10:52.245041 11260 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9b19d4d48fbe409e82e2b1baf07079dd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 05:10:52.245040 11400 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9b19d4d48fbe409e82e2b1baf07079dd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 05:10:52.246152 11635 tablet_bootstrap.cc:492] T 9b19d4d48fbe409e82e2b1baf07079dd P 017e3652ab544ab08232482357ea8d48: Bootstrap starting.
I20260621 05:10:52.246559 11635 tablet_bootstrap.cc:654] T 9b19d4d48fbe409e82e2b1baf07079dd P 017e3652ab544ab08232482357ea8d48: Neither blocks nor log segments found. Creating new log.
I20260621 05:10:52.246668 11637 tablet_bootstrap.cc:492] T 9b19d4d48fbe409e82e2b1baf07079dd P 6fa73262ba394487a5a54708f5de7075: Bootstrap starting.
I20260621 05:10:52.246670 11634 tablet_bootstrap.cc:492] T 9b19d4d48fbe409e82e2b1baf07079dd P eceee696a5ee4011b1264da66419eae9: Bootstrap starting.
I20260621 05:10:52.247067 11634 tablet_bootstrap.cc:654] T 9b19d4d48fbe409e82e2b1baf07079dd P eceee696a5ee4011b1264da66419eae9: Neither blocks nor log segments found. Creating new log.
I20260621 05:10:52.247097 11637 tablet_bootstrap.cc:654] T 9b19d4d48fbe409e82e2b1baf07079dd P 6fa73262ba394487a5a54708f5de7075: Neither blocks nor log segments found. Creating new log.
I20260621 05:10:52.247355 11635 tablet_bootstrap.cc:492] T 9b19d4d48fbe409e82e2b1baf07079dd P 017e3652ab544ab08232482357ea8d48: No bootstrap required, opened a new log
I20260621 05:10:52.247414 11635 ts_tablet_manager.cc:1397] T 9b19d4d48fbe409e82e2b1baf07079dd P 017e3652ab544ab08232482357ea8d48: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260621 05:10:52.247573 11635 raft_consensus.cc:348] T 9b19d4d48fbe409e82e2b1baf07079dd P 017e3652ab544ab08232482357ea8d48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fa73262ba394487a5a54708f5de7075" member_type: VOTER last_known_addr { host: "127.3.131.129" port: 38555 } } peers { permanent_uuid: "eceee696a5ee4011b1264da66419eae9" member_type: VOTER last_known_addr { host: "127.3.131.130" port: 36509 } } peers { permanent_uuid: "017e3652ab544ab08232482357ea8d48" member_type: VOTER last_known_addr { host: "127.3.131.131" port: 46349 } }
I20260621 05:10:52.247659 11635 raft_consensus.cc:374] T 9b19d4d48fbe409e82e2b1baf07079dd P 017e3652ab544ab08232482357ea8d48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 05:10:52.247679 11635 raft_consensus.cc:729] T 9b19d4d48fbe409e82e2b1baf07079dd P 017e3652ab544ab08232482357ea8d48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 017e3652ab544ab08232482357ea8d48, State: Initialized, Role: FOLLOWER
I20260621 05:10:52.247720 11635 consensus_queue.cc:260] T 9b19d4d48fbe409e82e2b1baf07079dd P 017e3652ab544ab08232482357ea8d48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fa73262ba394487a5a54708f5de7075" member_type: VOTER last_known_addr { host: "127.3.131.129" port: 38555 } } peers { permanent_uuid: "eceee696a5ee4011b1264da66419eae9" member_type: VOTER last_known_addr { host: "127.3.131.130" port: 36509 } } peers { permanent_uuid: "017e3652ab544ab08232482357ea8d48" member_type: VOTER last_known_addr { host: "127.3.131.131" port: 46349 } }
I20260621 05:10:52.247807 11635 ts_tablet_manager.cc:1428] T 9b19d4d48fbe409e82e2b1baf07079dd P 017e3652ab544ab08232482357ea8d48: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260621 05:10:52.247928 11637 tablet_bootstrap.cc:492] T 9b19d4d48fbe409e82e2b1baf07079dd P 6fa73262ba394487a5a54708f5de7075: No bootstrap required, opened a new log
I20260621 05:10:52.247980 11637 ts_tablet_manager.cc:1397] T 9b19d4d48fbe409e82e2b1baf07079dd P 6fa73262ba394487a5a54708f5de7075: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260621 05:10:52.248145 11637 raft_consensus.cc:348] T 9b19d4d48fbe409e82e2b1baf07079dd P 6fa73262ba394487a5a54708f5de7075 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fa73262ba394487a5a54708f5de7075" member_type: VOTER last_known_addr { host: "127.3.131.129" port: 38555 } } peers { permanent_uuid: "eceee696a5ee4011b1264da66419eae9" member_type: VOTER last_known_addr { host: "127.3.131.130" port: 36509 } } peers { permanent_uuid: "017e3652ab544ab08232482357ea8d48" member_type: VOTER last_known_addr { host: "127.3.131.131" port: 46349 } }
I20260621 05:10:52.248243 11637 raft_consensus.cc:374] T 9b19d4d48fbe409e82e2b1baf07079dd P 6fa73262ba394487a5a54708f5de7075 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 05:10:52.248282 11637 raft_consensus.cc:729] T 9b19d4d48fbe409e82e2b1baf07079dd P 6fa73262ba394487a5a54708f5de7075 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6fa73262ba394487a5a54708f5de7075, State: Initialized, Role: FOLLOWER
I20260621 05:10:52.248343 11637 consensus_queue.cc:260] T 9b19d4d48fbe409e82e2b1baf07079dd P 6fa73262ba394487a5a54708f5de7075 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fa73262ba394487a5a54708f5de7075" member_type: VOTER last_known_addr { host: "127.3.131.129" port: 38555 } } peers { permanent_uuid: "eceee696a5ee4011b1264da66419eae9" member_type: VOTER last_known_addr { host: "127.3.131.130" port: 36509 } } peers { permanent_uuid: "017e3652ab544ab08232482357ea8d48" member_type: VOTER last_known_addr { host: "127.3.131.131" port: 46349 } }
I20260621 05:10:52.248451 11637 ts_tablet_manager.cc:1428] T 9b19d4d48fbe409e82e2b1baf07079dd P 6fa73262ba394487a5a54708f5de7075: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260621 05:10:52.248489 11634 tablet_bootstrap.cc:492] T 9b19d4d48fbe409e82e2b1baf07079dd P eceee696a5ee4011b1264da66419eae9: No bootstrap required, opened a new log
I20260621 05:10:52.248544 11634 ts_tablet_manager.cc:1397] T 9b19d4d48fbe409e82e2b1baf07079dd P eceee696a5ee4011b1264da66419eae9: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260621 05:10:52.248575 11642 tablet_replica.cc:439] T 9b19d4d48fbe409e82e2b1baf07079dd P 017e3652ab544ab08232482357ea8d48: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fa73262ba394487a5a54708f5de7075" member_type: VOTER last_known_addr { host: "127.3.131.129" port: 38555 } } peers { permanent_uuid: "eceee696a5ee4011b1264da66419eae9" member_type: VOTER last_known_addr { host: "127.3.131.130" port: 36509 } } peers { permanent_uuid: "017e3652ab544ab08232482357ea8d48" member_type: VOTER last_known_addr { host: "127.3.131.131" port: 46349 } } }
I20260621 05:10:52.248600 11641 tablet_replica.cc:439] T 9b19d4d48fbe409e82e2b1baf07079dd P 6fa73262ba394487a5a54708f5de7075: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fa73262ba394487a5a54708f5de7075" member_type: VOTER last_known_addr { host: "127.3.131.129" port: 38555 } } peers { permanent_uuid: "eceee696a5ee4011b1264da66419eae9" member_type: VOTER last_known_addr { host: "127.3.131.130" port: 36509 } } peers { permanent_uuid: "017e3652ab544ab08232482357ea8d48" member_type: VOTER last_known_addr { host: "127.3.131.131" port: 46349 } } }
I20260621 05:10:52.248642 11634 raft_consensus.cc:348] T 9b19d4d48fbe409e82e2b1baf07079dd P eceee696a5ee4011b1264da66419eae9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fa73262ba394487a5a54708f5de7075" member_type: VOTER last_known_addr { host: "127.3.131.129" port: 38555 } } peers { permanent_uuid: "eceee696a5ee4011b1264da66419eae9" member_type: VOTER last_known_addr { host: "127.3.131.130" port: 36509 } } peers { permanent_uuid: "017e3652ab544ab08232482357ea8d48" member_type: VOTER last_known_addr { host: "127.3.131.131" port: 46349 } }
I20260621 05:10:52.248675 11641 tablet_replica.cc:442] T 9b19d4d48fbe409e82e2b1baf07079dd P 6fa73262ba394487a5a54708f5de7075: This TxnStatusTablet replica's current role is: FOLLOWER
I20260621 05:10:52.248668 11642 tablet_replica.cc:442] T 9b19d4d48fbe409e82e2b1baf07079dd P 017e3652ab544ab08232482357ea8d48: This TxnStatusTablet replica's current role is: FOLLOWER
I20260621 05:10:52.248699 11634 raft_consensus.cc:374] T 9b19d4d48fbe409e82e2b1baf07079dd P eceee696a5ee4011b1264da66419eae9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 05:10:52.248719 11634 raft_consensus.cc:729] T 9b19d4d48fbe409e82e2b1baf07079dd P eceee696a5ee4011b1264da66419eae9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eceee696a5ee4011b1264da66419eae9, State: Initialized, Role: FOLLOWER
I20260621 05:10:52.248761 11634 consensus_queue.cc:260] T 9b19d4d48fbe409e82e2b1baf07079dd P eceee696a5ee4011b1264da66419eae9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fa73262ba394487a5a54708f5de7075" member_type: VOTER last_known_addr { host: "127.3.131.129" port: 38555 } } peers { permanent_uuid: "eceee696a5ee4011b1264da66419eae9" member_type: VOTER last_known_addr { host: "127.3.131.130" port: 36509 } } peers { permanent_uuid: "017e3652ab544ab08232482357ea8d48" member_type: VOTER last_known_addr { host: "127.3.131.131" port: 46349 } }
I20260621 05:10:52.248842 11634 ts_tablet_manager.cc:1428] T 9b19d4d48fbe409e82e2b1baf07079dd P eceee696a5ee4011b1264da66419eae9: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260621 05:10:52.248909 11644 tablet_replica.cc:439] T 9b19d4d48fbe409e82e2b1baf07079dd P eceee696a5ee4011b1264da66419eae9: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fa73262ba394487a5a54708f5de7075" member_type: VOTER last_known_addr { host: "127.3.131.129" port: 38555 } } peers { permanent_uuid: "eceee696a5ee4011b1264da66419eae9" member_type: VOTER last_known_addr { host: "127.3.131.130" port: 36509 } } peers { permanent_uuid: "017e3652ab544ab08232482357ea8d48" member_type: VOTER last_known_addr { host: "127.3.131.131" port: 46349 } } }
I20260621 05:10:52.249003 11644 tablet_replica.cc:442] T 9b19d4d48fbe409e82e2b1baf07079dd P eceee696a5ee4011b1264da66419eae9: This TxnStatusTablet replica's current role is: FOLLOWER
I20260621 05:10:52.249500 11641 raft_consensus.cc:482] T 9b19d4d48fbe409e82e2b1baf07079dd P 6fa73262ba394487a5a54708f5de7075 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260621 05:10:52.249559 11641 raft_consensus.cc:504] T 9b19d4d48fbe409e82e2b1baf07079dd P 6fa73262ba394487a5a54708f5de7075 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fa73262ba394487a5a54708f5de7075" member_type: VOTER last_known_addr { host: "127.3.131.129" port: 38555 } } peers { permanent_uuid: "eceee696a5ee4011b1264da66419eae9" member_type: VOTER last_known_addr { host: "127.3.131.130" port: 36509 } } peers { permanent_uuid: "017e3652ab544ab08232482357ea8d48" member_type: VOTER last_known_addr { host: "127.3.131.131" port: 46349 } }
I20260621 05:10:52.249882 11641 leader_election.cc:288] T 9b19d4d48fbe409e82e2b1baf07079dd P 6fa73262ba394487a5a54708f5de7075 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers eceee696a5ee4011b1264da66419eae9 (127.3.131.130:36509), 017e3652ab544ab08232482357ea8d48 (127.3.131.131:46349)
I20260621 05:10:52.253369 11632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:10:52.250162 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.131.129:42719 (local address 127.3.131.131:46349)
0621 05:10:52.250233 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0621 05:10:52.250236 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0621 05:10:52.250317 (+ 81us) server_negotiation.cc:407] Connection header received
0621 05:10:52.250388 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:10:52.250389 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:10:52.250399 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:10:52.250452 (+ 53us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 05:10:52.250652 (+ 200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:10:52.251330 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:10:52.252504 (+ 1174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:10:52.252973 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:10:52.253033 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:10:52.253041 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0621 05:10:52.253249 (+ 208us) server_negotiation.cc:299] Negotiation successful
0621 05:10:52.253294 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260621 05:10:52.253391 11633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:10:52.250170 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.131.129:41159 (local address 127.3.131.130:36509)
0621 05:10:52.250428 (+ 258us) server_negotiation.cc:206] Beginning negotiation
0621 05:10:52.250430 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:10:52.250437 (+ 7us) server_negotiation.cc:407] Connection header received
0621 05:10:52.250459 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:10:52.250462 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:10:52.250478 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:10:52.250553 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 05:10:52.250770 (+ 217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:10:52.251290 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:10:52.252654 (+ 1364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:10:52.252995 (+ 341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:10:52.253090 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:10:52.253099 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0621 05:10:52.253287 (+ 188us) server_negotiation.cc:299] Negotiation successful
0621 05:10:52.253329 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":236}
I20260621 05:10:52.253372 11672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:10:52.250021 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.131.131:46349 (local address 127.3.131.129:42719)
0621 05:10:52.250229 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0621 05:10:52.250243 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0621 05:10:52.250317 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:10:52.250500 (+ 183us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:10:52.250501 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:10:52.250503 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 05:10:52.250584 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:10:52.250589 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:10:52.251430 (+ 841us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:10:52.251431 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:10:52.252429 (+ 998us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:10:52.252432 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:10:52.253060 (+ 628us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:10:52.253061 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:10:52.253148 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:10:52.253153 (+ 5us) client_negotiation.cc:769] Sending connection context
0621 05:10:52.253211 (+ 58us) client_negotiation.cc:240] Negotiation successful
0621 05:10:52.253265 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":191,"thread_start_us":59,"threads_started":1}
I20260621 05:10:52.253466 11673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:10:52.250075 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.131.130:36509 (local address 127.3.131.129:41159)
0621 05:10:52.250284 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0621 05:10:52.250293 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0621 05:10:52.250336 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:10:52.250558 (+ 222us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:10:52.250558 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:10:52.250560 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 05:10:52.250650 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:10:52.250652 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:10:52.251380 (+ 728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:10:52.251381 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:10:52.252565 (+ 1184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:10:52.252576 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:10:52.253115 (+ 539us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:10:52.253116 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:10:52.253174 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:10:52.253183 (+ 9us) client_negotiation.cc:769] Sending connection context
0621 05:10:52.253270 (+ 87us) client_negotiation.cc:240] Negotiation successful
0621 05:10:52.253312 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":196,"mutex_wait_us":30,"thread_start_us":103,"threads_started":1}
I20260621 05:10:52.254027 11420 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b19d4d48fbe409e82e2b1baf07079dd" candidate_uuid: "6fa73262ba394487a5a54708f5de7075" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eceee696a5ee4011b1264da66419eae9" is_pre_election: true
I20260621 05:10:52.254128 11560 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b19d4d48fbe409e82e2b1baf07079dd" candidate_uuid: "6fa73262ba394487a5a54708f5de7075" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "017e3652ab544ab08232482357ea8d48" is_pre_election: true
I20260621 05:10:52.254225 11560 raft_consensus.cc:2454] T 9b19d4d48fbe409e82e2b1baf07079dd P 017e3652ab544ab08232482357ea8d48 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6fa73262ba394487a5a54708f5de7075 in term 0.
I20260621 05:10:52.254184 11420 raft_consensus.cc:2454] T 9b19d4d48fbe409e82e2b1baf07079dd P eceee696a5ee4011b1264da66419eae9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6fa73262ba394487a5a54708f5de7075 in term 0.
I20260621 05:10:52.254411 11212 leader_election.cc:302] T 9b19d4d48fbe409e82e2b1baf07079dd P 6fa73262ba394487a5a54708f5de7075 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 017e3652ab544ab08232482357ea8d48, 6fa73262ba394487a5a54708f5de7075; no voters:
I20260621 05:10:52.254595 11641 raft_consensus.cc:2781] T 9b19d4d48fbe409e82e2b1baf07079dd P 6fa73262ba394487a5a54708f5de7075 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260621 05:10:52.254652 11641 raft_consensus.cc:482] T 9b19d4d48fbe409e82e2b1baf07079dd P 6fa73262ba394487a5a54708f5de7075 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260621 05:10:52.254688 11641 raft_consensus.cc:3037] T 9b19d4d48fbe409e82e2b1baf07079dd P 6fa73262ba394487a5a54708f5de7075 [term 0 FOLLOWER]: Advancing to term 1
I20260621 05:10:52.255630 11641 raft_consensus.cc:504] T 9b19d4d48fbe409e82e2b1baf07079dd P 6fa73262ba394487a5a54708f5de7075 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fa73262ba394487a5a54708f5de7075" member_type: VOTER last_known_addr { host: "127.3.131.129" port: 38555 } } peers { permanent_uuid: "eceee696a5ee4011b1264da66419eae9" member_type: VOTER last_known_addr { host: "127.3.131.130" port: 36509 } } peers { permanent_uuid: "017e3652ab544ab08232482357ea8d48" member_type: VOTER last_known_addr { host: "127.3.131.131" port: 46349 } }
I20260621 05:10:52.255802 11641 leader_election.cc:288] T 9b19d4d48fbe409e82e2b1baf07079dd P 6fa73262ba394487a5a54708f5de7075 [CANDIDATE]: Term 1 election: Requested vote from peers eceee696a5ee4011b1264da66419eae9 (127.3.131.130:36509), 017e3652ab544ab08232482357ea8d48 (127.3.131.131:46349)
I20260621 05:10:52.255956 11560 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b19d4d48fbe409e82e2b1baf07079dd" candidate_uuid: "6fa73262ba394487a5a54708f5de7075" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "017e3652ab544ab08232482357ea8d48"
I20260621 05:10:52.255951 11420 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b19d4d48fbe409e82e2b1baf07079dd" candidate_uuid: "6fa73262ba394487a5a54708f5de7075" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eceee696a5ee4011b1264da66419eae9"
I20260621 05:10:52.256027 11560 raft_consensus.cc:3037] T 9b19d4d48fbe409e82e2b1baf07079dd P 017e3652ab544ab08232482357ea8d48 [term 0 FOLLOWER]: Advancing to term 1
I20260621 05:10:52.256049 11420 raft_consensus.cc:3037] T 9b19d4d48fbe409e82e2b1baf07079dd P eceee696a5ee4011b1264da66419eae9 [term 0 FOLLOWER]: Advancing to term 1
I20260621 05:10:52.256706 11560 raft_consensus.cc:2454] T 9b19d4d48fbe409e82e2b1baf07079dd P 017e3652ab544ab08232482357ea8d48 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6fa73262ba394487a5a54708f5de7075 in term 1.
I20260621 05:10:52.256807 11420 raft_consensus.cc:2454] T 9b19d4d48fbe409e82e2b1baf07079dd P eceee696a5ee4011b1264da66419eae9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6fa73262ba394487a5a54708f5de7075 in term 1.
I20260621 05:10:52.256876 11212 leader_election.cc:302] T 9b19d4d48fbe409e82e2b1baf07079dd P 6fa73262ba394487a5a54708f5de7075 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 017e3652ab544ab08232482357ea8d48, 6fa73262ba394487a5a54708f5de7075; no voters:
I20260621 05:10:52.256999 11641 raft_consensus.cc:2781] T 9b19d4d48fbe409e82e2b1baf07079dd P 6fa73262ba394487a5a54708f5de7075 [term 1 FOLLOWER]: Leader election won for term 1
I20260621 05:10:52.257161 11641 raft_consensus.cc:686] T 9b19d4d48fbe409e82e2b1baf07079dd P 6fa73262ba394487a5a54708f5de7075 [term 1 LEADER]: Becoming Leader. State: Replica: 6fa73262ba394487a5a54708f5de7075, State: Running, Role: LEADER
I20260621 05:10:52.257248 11641 consensus_queue.cc:237] T 9b19d4d48fbe409e82e2b1baf07079dd P 6fa73262ba394487a5a54708f5de7075 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fa73262ba394487a5a54708f5de7075" member_type: VOTER last_known_addr { host: "127.3.131.129" port: 38555 } } peers { permanent_uuid: "eceee696a5ee4011b1264da66419eae9" member_type: VOTER last_known_addr { host: "127.3.131.130" port: 36509 } } peers { permanent_uuid: "017e3652ab544ab08232482357ea8d48" member_type: VOTER last_known_addr { host: "127.3.131.131" port: 46349 } }
I20260621 05:10:52.257618 11674 tablet_replica.cc:439] T 9b19d4d48fbe409e82e2b1baf07079dd P 6fa73262ba394487a5a54708f5de7075: TxnStatusTablet state changed. Reason: New leader 6fa73262ba394487a5a54708f5de7075. Latest consensus state: current_term: 1 leader_uuid: "6fa73262ba394487a5a54708f5de7075" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fa73262ba394487a5a54708f5de7075" member_type: VOTER last_known_addr { host: "127.3.131.129" port: 38555 } } peers { permanent_uuid: "eceee696a5ee4011b1264da66419eae9" member_type: VOTER last_known_addr { host: "127.3.131.130" port: 36509 } } peers { permanent_uuid: "017e3652ab544ab08232482357ea8d48" member_type: VOTER last_known_addr { host: "127.3.131.131" port: 46349 } } }
I20260621 05:10:52.257721 11674 tablet_replica.cc:442] T 9b19d4d48fbe409e82e2b1baf07079dd P 6fa73262ba394487a5a54708f5de7075: This TxnStatusTablet replica's current role is: LEADER
I20260621 05:10:52.257906 11676 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260621 05:10:52.258020 11676 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260621 05:10:52.258024 11121 catalog_manager.cc:5495] T 9b19d4d48fbe409e82e2b1baf07079dd P 6fa73262ba394487a5a54708f5de7075 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6fa73262ba394487a5a54708f5de7075 (127.3.131.129). New cstate: current_term: 1 leader_uuid: "6fa73262ba394487a5a54708f5de7075" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fa73262ba394487a5a54708f5de7075" member_type: VOTER last_known_addr { host: "127.3.131.129" port: 38555 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "eceee696a5ee4011b1264da66419eae9" member_type: VOTER last_known_addr { host: "127.3.131.130" port: 36509 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "017e3652ab544ab08232482357ea8d48" member_type: VOTER last_known_addr { host: "127.3.131.131" port: 46349 } health_report { overall_health: UNKNOWN } } }
I20260621 05:10:52.267970 11668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:10:52.264836 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.131.129:38555 (local address 127.0.0.1:41114)
0621 05:10:52.264915 (+ 79us) negotiation.cc:107] Waiting for socket to connect
0621 05:10:52.264928 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0621 05:10:52.264982 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:10:52.265205 (+ 223us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:10:52.265206 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:10:52.265207 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0621 05:10:52.265254 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:10:52.265256 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:10:52.266299 (+ 1043us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:10:52.266300 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:10:52.266872 (+ 572us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:10:52.266874 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:10:52.267441 (+ 567us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:10:52.267443 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:10:52.267501 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:10:52.267539 (+ 38us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0621 05:10:52.267841 (+ 302us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0621 05:10:52.267845 (+ 4us) client_negotiation.cc:769] Sending connection context
0621 05:10:52.267871 (+ 26us) client_negotiation.cc:240] Negotiation successful
0621 05:10:52.267911 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":40}
I20260621 05:10:52.267984 11631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:10:52.264950 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41114 (local address 127.3.131.129:38555)
0621 05:10:52.264993 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0621 05:10:52.264995 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:10:52.265005 (+ 10us) server_negotiation.cc:407] Connection header received
0621 05:10:52.265096 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:10:52.265096 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:10:52.265117 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:10:52.265207 (+ 90us) server_negotiation.cc:226] Negotiated authn=TOKEN
0621 05:10:52.265323 (+ 116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:10:52.266204 (+ 881us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:10:52.266990 (+ 786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:10:52.267361 (+ 371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:10:52.267440 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:10:52.267643 (+ 203us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0621 05:10:52.267775 (+ 132us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0621 05:10:52.267861 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0621 05:10:52.267880 (+ 19us) server_negotiation.cc:299] Negotiation successful
0621 05:10:52.267902 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260621 05:10:52.269275 11420 raft_consensus.cc:1261] T 9b19d4d48fbe409e82e2b1baf07079dd P eceee696a5ee4011b1264da66419eae9 [term 1 FOLLOWER]: Refusing update from remote peer 6fa73262ba394487a5a54708f5de7075: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 05:10:52.269289 11560 raft_consensus.cc:1261] T 9b19d4d48fbe409e82e2b1baf07079dd P 017e3652ab544ab08232482357ea8d48 [term 1 FOLLOWER]: Refusing update from remote peer 6fa73262ba394487a5a54708f5de7075: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 05:10:52.269550 11674 consensus_queue.cc:1035] T 9b19d4d48fbe409e82e2b1baf07079dd P 6fa73262ba394487a5a54708f5de7075 [LEADER]: Connected to new peer: Peer: permanent_uuid: "017e3652ab544ab08232482357ea8d48" member_type: VOTER last_known_addr { host: "127.3.131.131" port: 46349 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 05:10:52.269683 11641 consensus_queue.cc:1035] T 9b19d4d48fbe409e82e2b1baf07079dd P 6fa73262ba394487a5a54708f5de7075 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eceee696a5ee4011b1264da66419eae9" member_type: VOTER last_known_addr { host: "127.3.131.130" port: 36509 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 05:10:52.270292 11642 tablet_replica.cc:439] T 9b19d4d48fbe409e82e2b1baf07079dd P 017e3652ab544ab08232482357ea8d48: TxnStatusTablet state changed. Reason: New leader 6fa73262ba394487a5a54708f5de7075. Latest consensus state: current_term: 1 leader_uuid: "6fa73262ba394487a5a54708f5de7075" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fa73262ba394487a5a54708f5de7075" member_type: VOTER last_known_addr { host: "127.3.131.129" port: 38555 } } peers { permanent_uuid: "eceee696a5ee4011b1264da66419eae9" member_type: VOTER last_known_addr { host: "127.3.131.130" port: 36509 } } peers { permanent_uuid: "017e3652ab544ab08232482357ea8d48" member_type: VOTER last_known_addr { host: "127.3.131.131" port: 46349 } } }
I20260621 05:10:52.270388 11642 tablet_replica.cc:442] T 9b19d4d48fbe409e82e2b1baf07079dd P 017e3652ab544ab08232482357ea8d48: This TxnStatusTablet replica's current role is: FOLLOWER
I20260621 05:10:52.270347 11644 tablet_replica.cc:439] T 9b19d4d48fbe409e82e2b1baf07079dd P eceee696a5ee4011b1264da66419eae9: TxnStatusTablet state changed. Reason: New leader 6fa73262ba394487a5a54708f5de7075. Latest consensus state: current_term: 1 leader_uuid: "6fa73262ba394487a5a54708f5de7075" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fa73262ba394487a5a54708f5de7075" member_type: VOTER last_known_addr { host: "127.3.131.129" port: 38555 } } peers { permanent_uuid: "eceee696a5ee4011b1264da66419eae9" member_type: VOTER last_known_addr { host: "127.3.131.130" port: 36509 } } peers { permanent_uuid: "017e3652ab544ab08232482357ea8d48" member_type: VOTER last_known_addr { host: "127.3.131.131" port: 46349 } } }
I20260621 05:10:52.270431 11644 tablet_replica.cc:442] T 9b19d4d48fbe409e82e2b1baf07079dd P eceee696a5ee4011b1264da66419eae9: This TxnStatusTablet replica's current role is: FOLLOWER
I20260621 05:10:52.271364 11674 tablet_replica.cc:439] T 9b19d4d48fbe409e82e2b1baf07079dd P 6fa73262ba394487a5a54708f5de7075: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "6fa73262ba394487a5a54708f5de7075" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fa73262ba394487a5a54708f5de7075" member_type: VOTER last_known_addr { host: "127.3.131.129" port: 38555 } } peers { permanent_uuid: "eceee696a5ee4011b1264da66419eae9" member_type: VOTER last_known_addr { host: "127.3.131.130" port: 36509 } } peers { permanent_uuid: "017e3652ab544ab08232482357ea8d48" member_type: VOTER last_known_addr { host: "127.3.131.131" port: 46349 } } }
I20260621 05:10:52.271466 11674 tablet_replica.cc:442] T 9b19d4d48fbe409e82e2b1baf07079dd P 6fa73262ba394487a5a54708f5de7075: This TxnStatusTablet replica's current role is: LEADER
I20260621 05:10:52.271970 11647 mvcc.cc:205] Tried to move back new op lower bound from 7299148399692398592 to 7299148399646191616. Current Snapshot: MvccSnapshot[applied={T|T < 7299148399692398592}]
I20260621 05:10:52.272071 11644 tablet_replica.cc:439] T 9b19d4d48fbe409e82e2b1baf07079dd P eceee696a5ee4011b1264da66419eae9: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "6fa73262ba394487a5a54708f5de7075" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fa73262ba394487a5a54708f5de7075" member_type: VOTER last_known_addr { host: "127.3.131.129" port: 38555 } } peers { permanent_uuid: "eceee696a5ee4011b1264da66419eae9" member_type: VOTER last_known_addr { host: "127.3.131.130" port: 36509 } } peers { permanent_uuid: "017e3652ab544ab08232482357ea8d48" member_type: VOTER last_known_addr { host: "127.3.131.131" port: 46349 } } }
I20260621 05:10:52.272140 11644 tablet_replica.cc:442] T 9b19d4d48fbe409e82e2b1baf07079dd P eceee696a5ee4011b1264da66419eae9: This TxnStatusTablet replica's current role is: FOLLOWER
I20260621 05:10:52.271413 11681 tablet_replica.cc:439] T 9b19d4d48fbe409e82e2b1baf07079dd P 6fa73262ba394487a5a54708f5de7075: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "6fa73262ba394487a5a54708f5de7075" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fa73262ba394487a5a54708f5de7075" member_type: VOTER last_known_addr { host: "127.3.131.129" port: 38555 } } peers { permanent_uuid: "eceee696a5ee4011b1264da66419eae9" member_type: VOTER last_known_addr { host: "127.3.131.130" port: 36509 } } peers { permanent_uuid: "017e3652ab544ab08232482357ea8d48" member_type: VOTER last_known_addr { host: "127.3.131.131" port: 46349 } } }
I20260621 05:10:52.272173 11681 tablet_replica.cc:442] T 9b19d4d48fbe409e82e2b1baf07079dd P 6fa73262ba394487a5a54708f5de7075: This TxnStatusTablet replica's current role is: LEADER
I20260621 05:10:52.273171 11642 tablet_replica.cc:439] T 9b19d4d48fbe409e82e2b1baf07079dd P 017e3652ab544ab08232482357ea8d48: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "6fa73262ba394487a5a54708f5de7075" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fa73262ba394487a5a54708f5de7075" member_type: VOTER last_known_addr { host: "127.3.131.129" port: 38555 } } peers { permanent_uuid: "eceee696a5ee4011b1264da66419eae9" member_type: VOTER last_known_addr { host: "127.3.131.130" port: 36509 } } peers { permanent_uuid: "017e3652ab544ab08232482357ea8d48" member_type: VOTER last_known_addr { host: "127.3.131.131" port: 46349 } } }
I20260621 05:10:52.273273 11642 tablet_replica.cc:442] T 9b19d4d48fbe409e82e2b1baf07079dd P 017e3652ab544ab08232482357ea8d48: This TxnStatusTablet replica's current role is: FOLLOWER
I20260621 05:10:52.278402 11631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:10:52.275576 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.131.130:38575 (local address 127.3.131.129:38555)
0621 05:10:52.275619 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0621 05:10:52.275621 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:10:52.275720 (+ 99us) server_negotiation.cc:407] Connection header received
0621 05:10:52.275797 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:10:52.275798 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:10:52.275806 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:10:52.275851 (+ 45us) server_negotiation.cc:226] Negotiated authn=TOKEN
0621 05:10:52.275972 (+ 121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:10:52.276450 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:10:52.277714 (+ 1264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:10:52.277906 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:10:52.277969 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:10:52.278174 (+ 205us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0621 05:10:52.278201 (+ 27us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0621 05:10:52.278240 (+ 39us) server_negotiation.cc:1035] Waiting for connection context
0621 05:10:52.278329 (+ 89us) server_negotiation.cc:299] Negotiation successful
0621 05:10:52.278347 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260621 05:10:52.278481 11684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:10:52.275450 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.131.129:38555 (local address 127.3.131.130:38575)
0621 05:10:52.275637 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0621 05:10:52.275650 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0621 05:10:52.275733 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:10:52.275854 (+ 121us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:10:52.275855 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:10:52.275857 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0621 05:10:52.275913 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:10:52.275915 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:10:52.276528 (+ 613us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:10:52.276529 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:10:52.277604 (+ 1075us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:10:52.277607 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:10:52.277993 (+ 386us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:10:52.277994 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:10:52.278062 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:10:52.278083 (+ 21us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0621 05:10:52.278266 (+ 183us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0621 05:10:52.278268 (+ 2us) client_negotiation.cc:769] Sending connection context
0621 05:10:52.278328 (+ 60us) client_negotiation.cc:240] Negotiation successful
0621 05:10:52.278387 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":95,"threads_started":1}
I20260621 05:10:52.280818 11560 raft_consensus.cc:1261] T 3860238ba18542c182c642d4ae7d456a P 017e3652ab544ab08232482357ea8d48 [term 1 FOLLOWER]: Refusing update from remote peer eceee696a5ee4011b1264da66419eae9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 05:10:52.280983 11277 raft_consensus.cc:1261] T 3860238ba18542c182c642d4ae7d456a P 6fa73262ba394487a5a54708f5de7075 [term 1 FOLLOWER]: Refusing update from remote peer eceee696a5ee4011b1264da66419eae9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 05:10:52.281114 11640 consensus_queue.cc:1035] T 3860238ba18542c182c642d4ae7d456a P eceee696a5ee4011b1264da66419eae9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "017e3652ab544ab08232482357ea8d48" member_type: VOTER last_known_addr { host: "127.3.131.131" port: 46349 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 05:10:52.281211 11644 consensus_queue.cc:1035] T 3860238ba18542c182c642d4ae7d456a P eceee696a5ee4011b1264da66419eae9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6fa73262ba394487a5a54708f5de7075" member_type: VOTER last_known_addr { host: "127.3.131.129" port: 38555 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 05:10:52.291677 11689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:10:52.288155 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.131.130:36509 (local address 127.3.131.129:41427)
0621 05:10:52.288353 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0621 05:10:52.288365 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0621 05:10:52.288424 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:10:52.288544 (+ 120us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:10:52.288545 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:10:52.288546 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0621 05:10:52.288641 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:10:52.288644 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:10:52.289775 (+ 1131us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:10:52.289776 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:10:52.290587 (+ 811us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:10:52.290590 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:10:52.291111 (+ 521us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:10:52.291112 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:10:52.291225 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:10:52.291254 (+ 29us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0621 05:10:52.291478 (+ 224us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0621 05:10:52.291481 (+ 3us) client_negotiation.cc:769] Sending connection context
0621 05:10:52.291534 (+ 53us) client_negotiation.cc:240] Negotiation successful
0621 05:10:52.291580 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":161,"thread_start_us":76,"threads_started":1}
I20260621 05:10:52.291662 11633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:10:52.288217 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.131.129:41427 (local address 127.3.131.130:36509)
0621 05:10:52.288259 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0621 05:10:52.288261 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:10:52.288432 (+ 171us) server_negotiation.cc:407] Connection header received
0621 05:10:52.288462 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:10:52.288463 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:10:52.288473 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:10:52.288544 (+ 71us) server_negotiation.cc:226] Negotiated authn=TOKEN
0621 05:10:52.288699 (+ 155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:10:52.289531 (+ 832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:10:52.290671 (+ 1140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:10:52.290996 (+ 325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:10:52.291071 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:10:52.291341 (+ 270us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0621 05:10:52.291383 (+ 42us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0621 05:10:52.291440 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0621 05:10:52.291552 (+ 112us) server_negotiation.cc:299] Negotiation successful
0621 05:10:52.291578 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260621 05:10:52.300798 11122 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:57290:
table { table_name: "test-table" } modify_external_catalogs: true
I20260621 05:10:52.300935 11122 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:57290:
table { table_name: "test-table" } modify_external_catalogs: true
I20260621 05:10:52.301698 11122 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 65139302364d411cb10a6e712d783f40: Sending DeleteTablet for 3 replicas of tablet 3860238ba18542c182c642d4ae7d456a
I20260621 05:10:52.301975 11400 tablet_service.cc:1514] Processing DeleteTablet for tablet 3860238ba18542c182c642d4ae7d456a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 05:10:52 UTC) from {username='oryx', principal='oryx/127.3.131.190@KRBTEST.COM'} at 127.0.0.1:35264
I20260621 05:10:52.302017 11539 tablet_service.cc:1514] Processing DeleteTablet for tablet 3860238ba18542c182c642d4ae7d456a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 05:10:52 UTC) from {username='oryx', principal='oryx/127.3.131.190@KRBTEST.COM'} at 127.0.0.1:38122
I20260621 05:10:52.302088 11657 tablet_replica.cc:330] T 3860238ba18542c182c642d4ae7d456a P eceee696a5ee4011b1264da66419eae9: stopping tablet replica
I20260621 05:10:52.302140 11657 raft_consensus.cc:2229] T 3860238ba18542c182c642d4ae7d456a P eceee696a5ee4011b1264da66419eae9 [term 1 LEADER]: Raft consensus shutting down.
I20260621 05:10:52.302208 11657 raft_consensus.cc:2258] T 3860238ba18542c182c642d4ae7d456a P eceee696a5ee4011b1264da66419eae9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 05:10:52.302198 11658 tablet_replica.cc:330] T 3860238ba18542c182c642d4ae7d456a P 017e3652ab544ab08232482357ea8d48: stopping tablet replica
I20260621 05:10:52.302261 11658 raft_consensus.cc:2229] T 3860238ba18542c182c642d4ae7d456a P 017e3652ab544ab08232482357ea8d48 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260621 05:10:52.302305 11658 raft_consensus.cc:2258] T 3860238ba18542c182c642d4ae7d456a P 017e3652ab544ab08232482357ea8d48 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 05:10:52.302387 11260 tablet_service.cc:1514] Processing DeleteTablet for tablet 3860238ba18542c182c642d4ae7d456a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 05:10:52 UTC) from {username='oryx', principal='oryx/127.3.131.190@KRBTEST.COM'} at 127.0.0.1:41098
I20260621 05:10:52.302482 11656 tablet_replica.cc:330] T 3860238ba18542c182c642d4ae7d456a P 6fa73262ba394487a5a54708f5de7075: stopping tablet replica
I20260621 05:10:52.302528 11656 raft_consensus.cc:2229] T 3860238ba18542c182c642d4ae7d456a P 6fa73262ba394487a5a54708f5de7075 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260621 05:10:52.302565 11656 raft_consensus.cc:2258] T 3860238ba18542c182c642d4ae7d456a P 6fa73262ba394487a5a54708f5de7075 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 05:10:52.302590 11657 ts_tablet_manager.cc:1905] T 3860238ba18542c182c642d4ae7d456a P eceee696a5ee4011b1264da66419eae9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 05:10:52.302983 11658 ts_tablet_manager.cc:1905] T 3860238ba18542c182c642d4ae7d456a P 017e3652ab544ab08232482357ea8d48: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 05:10:52.305738 11658 ts_tablet_manager.cc:1918] T 3860238ba18542c182c642d4ae7d456a P 017e3652ab544ab08232482357ea8d48: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260621 05:10:52.305815 11658 log.cc:1196] T 3860238ba18542c182c642d4ae7d456a P 017e3652ab544ab08232482357ea8d48: Deleting WAL directory at /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782018616648578-3598-0/minicluster-data/ts-2/wal/wals/3860238ba18542c182c642d4ae7d456a
I20260621 05:10:52.306046 11658 ts_tablet_manager.cc:1939] T 3860238ba18542c182c642d4ae7d456a P 017e3652ab544ab08232482357ea8d48: Deleting consensus metadata
I20260621 05:10:52.306510 11656 ts_tablet_manager.cc:1905] T 3860238ba18542c182c642d4ae7d456a P 6fa73262ba394487a5a54708f5de7075: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 05:10:52.306962 11657 ts_tablet_manager.cc:1918] T 3860238ba18542c182c642d4ae7d456a P eceee696a5ee4011b1264da66419eae9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260621 05:10:52.307037 11657 log.cc:1196] T 3860238ba18542c182c642d4ae7d456a P eceee696a5ee4011b1264da66419eae9: Deleting WAL directory at /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782018616648578-3598-0/minicluster-data/ts-1/wal/wals/3860238ba18542c182c642d4ae7d456a
I20260621 05:10:52.307292 11657 ts_tablet_manager.cc:1939] T 3860238ba18542c182c642d4ae7d456a P eceee696a5ee4011b1264da66419eae9: Deleting consensus metadata
I20260621 05:10:52.308009 11109 catalog_manager.cc:4841] TS eceee696a5ee4011b1264da66419eae9 (127.3.131.130:36509): tablet 3860238ba18542c182c642d4ae7d456a (table test-table [id=e34454efe12843a6a7f72345f02c395e]) successfully deleted
I20260621 05:10:52.308207 11108 catalog_manager.cc:4841] TS 017e3652ab544ab08232482357ea8d48 (127.3.131.131:46349): tablet 3860238ba18542c182c642d4ae7d456a (table test-table [id=e34454efe12843a6a7f72345f02c395e]) successfully deleted
I20260621 05:10:52.308714 11656 ts_tablet_manager.cc:1918] T 3860238ba18542c182c642d4ae7d456a P 6fa73262ba394487a5a54708f5de7075: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260621 05:10:52.308782 11656 log.cc:1196] T 3860238ba18542c182c642d4ae7d456a P 6fa73262ba394487a5a54708f5de7075: Deleting WAL directory at /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782018616648578-3598-0/minicluster-data/ts-0/wal/wals/3860238ba18542c182c642d4ae7d456a
I20260621 05:10:52.309000 11656 ts_tablet_manager.cc:1939] T 3860238ba18542c182c642d4ae7d456a P 6fa73262ba394487a5a54708f5de7075: Deleting consensus metadata
I20260621 05:10:52.309548 11107 catalog_manager.cc:4841] TS 6fa73262ba394487a5a54708f5de7075 (127.3.131.129:38555): tablet 3860238ba18542c182c642d4ae7d456a (table test-table [id=e34454efe12843a6a7f72345f02c395e]) successfully deleted
Jun 21 05:10:52 dist-test-slave-vrq1 krb5kdc[11078](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782018652, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 21 05:10:52 dist-test-slave-vrq1 krb5kdc[11078](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.3.131.190@KRBTEST.COM, Server not found in Kerberos database
Jun 21 05:10:52 dist-test-slave-vrq1 krb5kdc[11078](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.3.131.190@KRBTEST.COM, Server not found in Kerberos database
W20260621 05:10:52.333081 11174 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 05:10:52.326187 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57310 (local address 127.3.131.190:33031)
0621 05:10:52.326258 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0621 05:10:52.326261 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0621 05:10:52.326424 (+ 163us) server_negotiation.cc:407] Connection header received
0621 05:10:52.326502 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:10:52.326504 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:10:52.326516 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:10:52.326581 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:10:52.327302 (+ 721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:10:52.328044 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:10:52.329012 (+ 968us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:10:52.329359 (+ 347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:10:52.329437 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:10:52.333006 (+ 3569us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57310: BlockingRecv error: recv got EOF from 127.0.0.1:57310 (error 108)
Metrics: {"server-negotiator.queue_time_us":37}
Jun 21 05:10:52 dist-test-slave-vrq1 krb5kdc[11078](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782018652, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.3.131.190@KRBTEST.COM
I20260621 05:10:52.343011 11174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:10:52.334060 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57322 (local address 127.3.131.190:33031)
0621 05:10:52.334113 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0621 05:10:52.334116 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0621 05:10:52.334357 (+ 241us) server_negotiation.cc:407] Connection header received
0621 05:10:52.334409 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:10:52.334410 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:10:52.334422 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:10:52.334480 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:10:52.334956 (+ 476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:10:52.335774 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:10:52.336577 (+ 803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:10:52.336877 (+ 300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:10:52.336954 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:10:52.339559 (+ 2605us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:10:52.339560 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:10:52.339561 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:10:52.339569 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:10:52.341558 (+ 1989us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:10:52.342107 (+ 549us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:10:52.342108 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:10:52.342109 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:10:52.342185 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:10:52.342406 (+ 221us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:10:52.342407 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:10:52.342407 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:10:52.342549 (+ 142us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:10:52.342661 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0621 05:10:52.342781 (+ 120us) server_negotiation.cc:299] Negotiation successful
0621 05:10:52.342879 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260621 05:10:52.344976 11122 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57322:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260621 05:10:52.345204 11122 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260621 05:10:52.346810 11400 tablet_service.cc:1467] Processing CreateTablet for tablet a1e067fcaf7749d39d931dbc49f28250 (DEFAULT_TABLE table=test-table [id=0916a34aeaf0400c80fc53514c8aa7d4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 05:10:52.346901 11539 tablet_service.cc:1467] Processing CreateTablet for tablet a1e067fcaf7749d39d931dbc49f28250 (DEFAULT_TABLE table=test-table [id=0916a34aeaf0400c80fc53514c8aa7d4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 05:10:52.346990 11400 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a1e067fcaf7749d39d931dbc49f28250. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 05:10:52.347013 11539 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a1e067fcaf7749d39d931dbc49f28250. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 05:10:52.347307 11260 tablet_service.cc:1467] Processing CreateTablet for tablet a1e067fcaf7749d39d931dbc49f28250 (DEFAULT_TABLE table=test-table [id=0916a34aeaf0400c80fc53514c8aa7d4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 05:10:52.347424 11260 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a1e067fcaf7749d39d931dbc49f28250. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 05:10:52.348490 11635 tablet_bootstrap.cc:492] T a1e067fcaf7749d39d931dbc49f28250 P 017e3652ab544ab08232482357ea8d48: Bootstrap starting.
I20260621 05:10:52.348839 11637 tablet_bootstrap.cc:492] T a1e067fcaf7749d39d931dbc49f28250 P 6fa73262ba394487a5a54708f5de7075: Bootstrap starting.
I20260621 05:10:52.348907 11635 tablet_bootstrap.cc:654] T a1e067fcaf7749d39d931dbc49f28250 P 017e3652ab544ab08232482357ea8d48: Neither blocks nor log segments found. Creating new log.
I20260621 05:10:52.349188 11637 tablet_bootstrap.cc:654] T a1e067fcaf7749d39d931dbc49f28250 P 6fa73262ba394487a5a54708f5de7075: Neither blocks nor log segments found. Creating new log.
I20260621 05:10:52.349783 11637 tablet_bootstrap.cc:492] T a1e067fcaf7749d39d931dbc49f28250 P 6fa73262ba394487a5a54708f5de7075: No bootstrap required, opened a new log
I20260621 05:10:52.349845 11637 ts_tablet_manager.cc:1397] T a1e067fcaf7749d39d931dbc49f28250 P 6fa73262ba394487a5a54708f5de7075: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260621 05:10:52.349967 11637 raft_consensus.cc:348] T a1e067fcaf7749d39d931dbc49f28250 P 6fa73262ba394487a5a54708f5de7075 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fa73262ba394487a5a54708f5de7075" member_type: VOTER last_known_addr { host: "127.3.131.129" port: 38555 } } peers { permanent_uuid: "eceee696a5ee4011b1264da66419eae9" member_type: VOTER last_known_addr { host: "127.3.131.130" port: 36509 } } peers { permanent_uuid: "017e3652ab544ab08232482357ea8d48" member_type: VOTER last_known_addr { host: "127.3.131.131" port: 46349 } }
I20260621 05:10:52.350054 11637 raft_consensus.cc:374] T a1e067fcaf7749d39d931dbc49f28250 P 6fa73262ba394487a5a54708f5de7075 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 05:10:52.350078 11637 raft_consensus.cc:729] T a1e067fcaf7749d39d931dbc49f28250 P 6fa73262ba394487a5a54708f5de7075 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6fa73262ba394487a5a54708f5de7075, State: Initialized, Role: FOLLOWER
I20260621 05:10:52.350145 11637 consensus_queue.cc:260] T a1e067fcaf7749d39d931dbc49f28250 P 6fa73262ba394487a5a54708f5de7075 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fa73262ba394487a5a54708f5de7075" member_type: VOTER last_known_addr { host: "127.3.131.129" port: 38555 } } peers { permanent_uuid: "eceee696a5ee4011b1264da66419eae9" member_type: VOTER last_known_addr { host: "127.3.131.130" port: 36509 } } peers { permanent_uuid: "017e3652ab544ab08232482357ea8d48" member_type: VOTER last_known_addr { host: "127.3.131.131" port: 46349 } }
I20260621 05:10:52.350203 11634 tablet_bootstrap.cc:492] T a1e067fcaf7749d39d931dbc49f28250 P eceee696a5ee4011b1264da66419eae9: Bootstrap starting.
I20260621 05:10:52.350247 11637 ts_tablet_manager.cc:1428] T a1e067fcaf7749d39d931dbc49f28250 P 6fa73262ba394487a5a54708f5de7075: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260621 05:10:52.350531 11634 tablet_bootstrap.cc:654] T a1e067fcaf7749d39d931dbc49f28250 P eceee696a5ee4011b1264da66419eae9: Neither blocks nor log segments found. Creating new log.
I20260621 05:10:52.350728 11635 tablet_bootstrap.cc:492] T a1e067fcaf7749d39d931dbc49f28250 P 017e3652ab544ab08232482357ea8d48: No bootstrap required, opened a new log
I20260621 05:10:52.350775 11635 ts_tablet_manager.cc:1397] T a1e067fcaf7749d39d931dbc49f28250 P 017e3652ab544ab08232482357ea8d48: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260621 05:10:52.350919 11635 raft_consensus.cc:348] T a1e067fcaf7749d39d931dbc49f28250 P 017e3652ab544ab08232482357ea8d48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fa73262ba394487a5a54708f5de7075" member_type: VOTER last_known_addr { host: "127.3.131.129" port: 38555 } } peers { permanent_uuid: "eceee696a5ee4011b1264da66419eae9" member_type: VOTER last_known_addr { host: "127.3.131.130" port: 36509 } } peers { permanent_uuid: "017e3652ab544ab08232482357ea8d48" member_type: VOTER last_known_addr { host: "127.3.131.131" port: 46349 } }
I20260621 05:10:52.350992 11635 raft_consensus.cc:374] T a1e067fcaf7749d39d931dbc49f28250 P 017e3652ab544ab08232482357ea8d48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 05:10:52.351009 11635 raft_consensus.cc:729] T a1e067fcaf7749d39d931dbc49f28250 P 017e3652ab544ab08232482357ea8d48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 017e3652ab544ab08232482357ea8d48, State: Initialized, Role: FOLLOWER
I20260621 05:10:52.351058 11635 consensus_queue.cc:260] T a1e067fcaf7749d39d931dbc49f28250 P 017e3652ab544ab08232482357ea8d48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fa73262ba394487a5a54708f5de7075" member_type: VOTER last_known_addr { host: "127.3.131.129" port: 38555 } } peers { permanent_uuid: "eceee696a5ee4011b1264da66419eae9" member_type: VOTER last_known_addr { host: "127.3.131.130" port: 36509 } } peers { permanent_uuid: "017e3652ab544ab08232482357ea8d48" member_type: VOTER last_known_addr { host: "127.3.131.131" port: 46349 } }
I20260621 05:10:52.351155 11635 ts_tablet_manager.cc:1428] T a1e067fcaf7749d39d931dbc49f28250 P 017e3652ab544ab08232482357ea8d48: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260621 05:10:52.351603 11634 tablet_bootstrap.cc:492] T a1e067fcaf7749d39d931dbc49f28250 P eceee696a5ee4011b1264da66419eae9: No bootstrap required, opened a new log
I20260621 05:10:52.351658 11634 ts_tablet_manager.cc:1397] T a1e067fcaf7749d39d931dbc49f28250 P eceee696a5ee4011b1264da66419eae9: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260621 05:10:52.351807 11634 raft_consensus.cc:348] T a1e067fcaf7749d39d931dbc49f28250 P eceee696a5ee4011b1264da66419eae9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fa73262ba394487a5a54708f5de7075" member_type: VOTER last_known_addr { host: "127.3.131.129" port: 38555 } } peers { permanent_uuid: "eceee696a5ee4011b1264da66419eae9" member_type: VOTER last_known_addr { host: "127.3.131.130" port: 36509 } } peers { permanent_uuid: "017e3652ab544ab08232482357ea8d48" member_type: VOTER last_known_addr { host: "127.3.131.131" port: 46349 } }
I20260621 05:10:52.351882 11634 raft_consensus.cc:374] T a1e067fcaf7749d39d931dbc49f28250 P eceee696a5ee4011b1264da66419eae9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 05:10:52.351914 11634 raft_consensus.cc:729] T a1e067fcaf7749d39d931dbc49f28250 P eceee696a5ee4011b1264da66419eae9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eceee696a5ee4011b1264da66419eae9, State: Initialized, Role: FOLLOWER
I20260621 05:10:52.351958 11634 consensus_queue.cc:260] T a1e067fcaf7749d39d931dbc49f28250 P eceee696a5ee4011b1264da66419eae9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fa73262ba394487a5a54708f5de7075" member_type: VOTER last_known_addr { host: "127.3.131.129" port: 38555 } } peers { permanent_uuid: "eceee696a5ee4011b1264da66419eae9" member_type: VOTER last_known_addr { host: "127.3.131.130" port: 36509 } } peers { permanent_uuid: "017e3652ab544ab08232482357ea8d48" member_type: VOTER last_known_addr { host: "127.3.131.131" port: 46349 } }
I20260621 05:10:52.352054 11634 ts_tablet_manager.cc:1428] T a1e067fcaf7749d39d931dbc49f28250 P eceee696a5ee4011b1264da66419eae9: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260621 05:10:52.424734 11642 raft_consensus.cc:482] T a1e067fcaf7749d39d931dbc49f28250 P 017e3652ab544ab08232482357ea8d48 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260621 05:10:52.424892 11642 raft_consensus.cc:504] T a1e067fcaf7749d39d931dbc49f28250 P 017e3652ab544ab08232482357ea8d48 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fa73262ba394487a5a54708f5de7075" member_type: VOTER last_known_addr { host: "127.3.131.129" port: 38555 } } peers { permanent_uuid: "eceee696a5ee4011b1264da66419eae9" member_type: VOTER last_known_addr { host: "127.3.131.130" port: 36509 } } peers { permanent_uuid: "017e3652ab544ab08232482357ea8d48" member_type: VOTER last_known_addr { host: "127.3.131.131" port: 46349 } }
I20260621 05:10:52.425420 11642 leader_election.cc:288] T a1e067fcaf7749d39d931dbc49f28250 P 017e3652ab544ab08232482357ea8d48 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6fa73262ba394487a5a54708f5de7075 (127.3.131.129:38555), eceee696a5ee4011b1264da66419eae9 (127.3.131.130:36509)
I20260621 05:10:52.429759 11633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:10:52.425889 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.131.131:51479 (local address 127.3.131.130:36509)
0621 05:10:52.425969 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0621 05:10:52.425971 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:10:52.425979 (+ 8us) server_negotiation.cc:407] Connection header received
0621 05:10:52.426016 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:10:52.426017 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:10:52.426028 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:10:52.426100 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 05:10:52.426324 (+ 224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:10:52.426989 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:10:52.428549 (+ 1560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:10:52.429217 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:10:52.429314 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:10:52.429325 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0621 05:10:52.429630 (+ 305us) server_negotiation.cc:299] Negotiation successful
0621 05:10:52.429681 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260621 05:10:52.429783 11706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:10:52.425668 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.131.129:38555 (local address 127.3.131.131:43701)
0621 05:10:52.425790 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0621 05:10:52.425798 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0621 05:10:52.425858 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:10:52.426307 (+ 449us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:10:52.426308 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:10:52.426312 (+ 4us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 05:10:52.426422 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:10:52.426424 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:10:52.427567 (+ 1143us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:10:52.427568 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:10:52.428632 (+ 1064us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:10:52.428633 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:10:52.429417 (+ 784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:10:52.429418 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:10:52.429506 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:10:52.429509 (+ 3us) client_negotiation.cc:769] Sending connection context
0621 05:10:52.429616 (+ 107us) client_negotiation.cc:240] Negotiation successful
0621 05:10:52.429670 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":113,"mutex_wait_us":1,"thread_start_us":53,"threads_started":1}
I20260621 05:10:52.429756 11631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:10:52.425838 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.131.131:43701 (local address 127.3.131.129:38555)
0621 05:10:52.425934 (+ 96us) server_negotiation.cc:206] Beginning negotiation
0621 05:10:52.425936 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:10:52.425946 (+ 10us) server_negotiation.cc:407] Connection header received
0621 05:10:52.425978 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:10:52.425979 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:10:52.425990 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:10:52.426233 (+ 243us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 05:10:52.426570 (+ 337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:10:52.427457 (+ 887us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:10:52.428736 (+ 1279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:10:52.429275 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:10:52.429351 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:10:52.429360 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0621 05:10:52.429633 (+ 273us) server_negotiation.cc:299] Negotiation successful
0621 05:10:52.429677 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260621 05:10:52.430048 11707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:10:52.425617 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.131.130:36509 (local address 127.3.131.131:51479)
0621 05:10:52.425771 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0621 05:10:52.425787 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0621 05:10:52.425858 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:10:52.426166 (+ 308us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:10:52.426167 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:10:52.426169 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 05:10:52.426244 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:10:52.426246 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:10:52.427077 (+ 831us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:10:52.427078 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:10:52.428446 (+ 1368us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:10:52.428449 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:10:52.429410 (+ 961us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:10:52.429412 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:10:52.429506 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:10:52.429512 (+ 6us) client_negotiation.cc:769] Sending connection context
0621 05:10:52.429912 (+ 400us) client_negotiation.cc:240] Negotiation successful
0621 05:10:52.429964 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":133,"mutex_wait_us":1,"thread_start_us":85,"threads_started":1}
I20260621 05:10:52.430528 11277 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a1e067fcaf7749d39d931dbc49f28250" candidate_uuid: "017e3652ab544ab08232482357ea8d48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6fa73262ba394487a5a54708f5de7075" is_pre_election: true
I20260621 05:10:52.430649 11277 raft_consensus.cc:2454] T a1e067fcaf7749d39d931dbc49f28250 P 6fa73262ba394487a5a54708f5de7075 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 017e3652ab544ab08232482357ea8d48 in term 0.
I20260621 05:10:52.430629 11420 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a1e067fcaf7749d39d931dbc49f28250" candidate_uuid: "017e3652ab544ab08232482357ea8d48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eceee696a5ee4011b1264da66419eae9" is_pre_election: true
I20260621 05:10:52.430714 11420 raft_consensus.cc:2454] T a1e067fcaf7749d39d931dbc49f28250 P eceee696a5ee4011b1264da66419eae9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 017e3652ab544ab08232482357ea8d48 in term 0.
I20260621 05:10:52.430860 11491 leader_election.cc:302] T a1e067fcaf7749d39d931dbc49f28250 P 017e3652ab544ab08232482357ea8d48 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 017e3652ab544ab08232482357ea8d48, 6fa73262ba394487a5a54708f5de7075; no voters:
I20260621 05:10:52.431021 11642 raft_consensus.cc:2781] T a1e067fcaf7749d39d931dbc49f28250 P 017e3652ab544ab08232482357ea8d48 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260621 05:10:52.431069 11642 raft_consensus.cc:482] T a1e067fcaf7749d39d931dbc49f28250 P 017e3652ab544ab08232482357ea8d48 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260621 05:10:52.431102 11642 raft_consensus.cc:3037] T a1e067fcaf7749d39d931dbc49f28250 P 017e3652ab544ab08232482357ea8d48 [term 0 FOLLOWER]: Advancing to term 1
I20260621 05:10:52.431865 11642 raft_consensus.cc:504] T a1e067fcaf7749d39d931dbc49f28250 P 017e3652ab544ab08232482357ea8d48 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fa73262ba394487a5a54708f5de7075" member_type: VOTER last_known_addr { host: "127.3.131.129" port: 38555 } } peers { permanent_uuid: "eceee696a5ee4011b1264da66419eae9" member_type: VOTER last_known_addr { host: "127.3.131.130" port: 36509 } } peers { permanent_uuid: "017e3652ab544ab08232482357ea8d48" member_type: VOTER last_known_addr { host: "127.3.131.131" port: 46349 } }
I20260621 05:10:52.432060 11642 leader_election.cc:288] T a1e067fcaf7749d39d931dbc49f28250 P 017e3652ab544ab08232482357ea8d48 [CANDIDATE]: Term 1 election: Requested vote from peers 6fa73262ba394487a5a54708f5de7075 (127.3.131.129:38555), eceee696a5ee4011b1264da66419eae9 (127.3.131.130:36509)
I20260621 05:10:52.432207 11420 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a1e067fcaf7749d39d931dbc49f28250" candidate_uuid: "017e3652ab544ab08232482357ea8d48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eceee696a5ee4011b1264da66419eae9"
I20260621 05:10:52.432299 11420 raft_consensus.cc:3037] T a1e067fcaf7749d39d931dbc49f28250 P eceee696a5ee4011b1264da66419eae9 [term 0 FOLLOWER]: Advancing to term 1
I20260621 05:10:52.432372 11277 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a1e067fcaf7749d39d931dbc49f28250" candidate_uuid: "017e3652ab544ab08232482357ea8d48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6fa73262ba394487a5a54708f5de7075"
I20260621 05:10:52.432454 11277 raft_consensus.cc:3037] T a1e067fcaf7749d39d931dbc49f28250 P 6fa73262ba394487a5a54708f5de7075 [term 0 FOLLOWER]: Advancing to term 1
I20260621 05:10:52.433207 11420 raft_consensus.cc:2454] T a1e067fcaf7749d39d931dbc49f28250 P eceee696a5ee4011b1264da66419eae9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 017e3652ab544ab08232482357ea8d48 in term 1.
I20260621 05:10:52.433234 11277 raft_consensus.cc:2454] T a1e067fcaf7749d39d931dbc49f28250 P 6fa73262ba394487a5a54708f5de7075 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 017e3652ab544ab08232482357ea8d48 in term 1.
I20260621 05:10:52.433389 11491 leader_election.cc:302] T a1e067fcaf7749d39d931dbc49f28250 P 017e3652ab544ab08232482357ea8d48 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 017e3652ab544ab08232482357ea8d48, 6fa73262ba394487a5a54708f5de7075; no voters:
I20260621 05:10:52.433525 11642 raft_consensus.cc:2781] T a1e067fcaf7749d39d931dbc49f28250 P 017e3652ab544ab08232482357ea8d48 [term 1 FOLLOWER]: Leader election won for term 1
I20260621 05:10:52.433699 11642 raft_consensus.cc:686] T a1e067fcaf7749d39d931dbc49f28250 P 017e3652ab544ab08232482357ea8d48 [term 1 LEADER]: Becoming Leader. State: Replica: 017e3652ab544ab08232482357ea8d48, State: Running, Role: LEADER
I20260621 05:10:52.433795 11642 consensus_queue.cc:237] T a1e067fcaf7749d39d931dbc49f28250 P 017e3652ab544ab08232482357ea8d48 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fa73262ba394487a5a54708f5de7075" member_type: VOTER last_known_addr { host: "127.3.131.129" port: 38555 } } peers { permanent_uuid: "eceee696a5ee4011b1264da66419eae9" member_type: VOTER last_known_addr { host: "127.3.131.130" port: 36509 } } peers { permanent_uuid: "017e3652ab544ab08232482357ea8d48" member_type: VOTER last_known_addr { host: "127.3.131.131" port: 46349 } }
I20260621 05:10:52.434486 11122 catalog_manager.cc:5495] T a1e067fcaf7749d39d931dbc49f28250 P 017e3652ab544ab08232482357ea8d48 reported cstate change: term changed from 0 to 1, leader changed from <none> to 017e3652ab544ab08232482357ea8d48 (127.3.131.131). New cstate: current_term: 1 leader_uuid: "017e3652ab544ab08232482357ea8d48" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fa73262ba394487a5a54708f5de7075" member_type: VOTER last_known_addr { host: "127.3.131.129" port: 38555 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "eceee696a5ee4011b1264da66419eae9" member_type: VOTER last_known_addr { host: "127.3.131.130" port: 36509 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "017e3652ab544ab08232482357ea8d48" member_type: VOTER last_known_addr { host: "127.3.131.131" port: 46349 } health_report { overall_health: HEALTHY } } }
I20260621 05:10:52.450222 11632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:10:52.447670 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38136 (local address 127.3.131.131:46349)
0621 05:10:52.447723 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0621 05:10:52.447725 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:10:52.447735 (+ 10us) server_negotiation.cc:407] Connection header received
0621 05:10:52.447854 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:10:52.447855 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:10:52.447884 (+ 29us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:10:52.447971 (+ 87us) server_negotiation.cc:226] Negotiated authn=TOKEN
0621 05:10:52.448158 (+ 187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:10:52.448659 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:10:52.449410 (+ 751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:10:52.449608 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:10:52.449687 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:10:52.449867 (+ 180us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0621 05:10:52.449974 (+ 107us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0621 05:10:52.450063 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0621 05:10:52.450102 (+ 39us) server_negotiation.cc:299] Negotiation successful
0621 05:10:52.450125 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260621 05:10:52.451349 11420 raft_consensus.cc:1261] T a1e067fcaf7749d39d931dbc49f28250 P eceee696a5ee4011b1264da66419eae9 [term 1 FOLLOWER]: Refusing update from remote peer 017e3652ab544ab08232482357ea8d48: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 05:10:52.451447 11277 raft_consensus.cc:1261] T a1e067fcaf7749d39d931dbc49f28250 P 6fa73262ba394487a5a54708f5de7075 [term 1 FOLLOWER]: Refusing update from remote peer 017e3652ab544ab08232482357ea8d48: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 05:10:52.451684 11708 consensus_queue.cc:1035] T a1e067fcaf7749d39d931dbc49f28250 P 017e3652ab544ab08232482357ea8d48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eceee696a5ee4011b1264da66419eae9" member_type: VOTER last_known_addr { host: "127.3.131.130" port: 36509 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 05:10:52.451848 11642 consensus_queue.cc:1035] T a1e067fcaf7749d39d931dbc49f28250 P 017e3652ab544ab08232482357ea8d48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6fa73262ba394487a5a54708f5de7075" member_type: VOTER last_known_addr { host: "127.3.131.129" port: 38555 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 05:10:52.455391 11122 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57322:
table { table_name: "test-table" } modify_external_catalogs: true
I20260621 05:10:52.455524 11122 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57322:
table { table_name: "test-table" } modify_external_catalogs: true
I20260621 05:10:52.456202 11122 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 65139302364d411cb10a6e712d783f40: Sending DeleteTablet for 3 replicas of tablet a1e067fcaf7749d39d931dbc49f28250
I20260621 05:10:52.456502 11539 tablet_service.cc:1514] Processing DeleteTablet for tablet a1e067fcaf7749d39d931dbc49f28250 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 05:10:52 UTC) from {username='oryx', principal='oryx/127.3.131.190@KRBTEST.COM'} at 127.0.0.1:38122
I20260621 05:10:52.456501 11400 tablet_service.cc:1514] Processing DeleteTablet for tablet a1e067fcaf7749d39d931dbc49f28250 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 05:10:52 UTC) from {username='oryx', principal='oryx/127.3.131.190@KRBTEST.COM'} at 127.0.0.1:35264
I20260621 05:10:52.456594 11657 tablet_replica.cc:330] T a1e067fcaf7749d39d931dbc49f28250 P eceee696a5ee4011b1264da66419eae9: stopping tablet replica
I20260621 05:10:52.456594 11658 tablet_replica.cc:330] T a1e067fcaf7749d39d931dbc49f28250 P 017e3652ab544ab08232482357ea8d48: stopping tablet replica
I20260621 05:10:52.456635 11657 raft_consensus.cc:2229] T a1e067fcaf7749d39d931dbc49f28250 P eceee696a5ee4011b1264da66419eae9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260621 05:10:52.456637 11658 raft_consensus.cc:2229] T a1e067fcaf7749d39d931dbc49f28250 P 017e3652ab544ab08232482357ea8d48 [term 1 LEADER]: Raft consensus shutting down.
I20260621 05:10:52.456670 11657 pending_rounds.cc:62] T a1e067fcaf7749d39d931dbc49f28250 P eceee696a5ee4011b1264da66419eae9: Trying to abort 2 pending ops.
I20260621 05:10:52.456743 11658 raft_consensus.cc:2258] T a1e067fcaf7749d39d931dbc49f28250 P 017e3652ab544ab08232482357ea8d48 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 05:10:52.456691 11657 pending_rounds.cc:68] T a1e067fcaf7749d39d931dbc49f28250 P eceee696a5ee4011b1264da66419eae9: Aborting op as it isn't in flight: id { term: 1 index: 2 } timestamp: 7299148400438448128 op_type: WRITE_OP write_request { tablet_id: "a1e067fcaf7749d39d931dbc49f28250" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } columns { name: "val" type: INT32 is_key: false is_nullable: true immutable: false } } row_operations { rows: "<redacted>""\001\003\00090\000\0001\324\000\000" indirect_data: "<redacted>""" } external_consistency_mode: CLIENT_PROPAGATED authz_token { token_data: "\010\210\347\335\321\006\"8\n\ntest-admin\022*\n 0916a34aeaf0400c80fc53514c8aa7d4\020\001\030\001 \001(\001" signature: "<redacted>"":G\003\360\341T\373\\\307\242\232\222\023\363\251\205\030\345\264\010\036\n\335\003\020F\3770\203\352\365\226\363Z\323\023\257A[\223\220`\217J\211\023K\270\217\014\374\354H\203\004\021\227?\014\323E\273\241\312" signing_key_seq_num: 0 } } request_id { client_id: "7ffe8fc9a5614b499e185989f9d1eaa0" seq_no: 0 first_incomplete_seq_no: 0 attempt_no: 0 }
I20260621 05:10:52.456929 11657 pending_rounds.cc:68] T a1e067fcaf7749d39d931dbc49f28250 P eceee696a5ee4011b1264da66419eae9: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7299148400369405952 op_type: NO_OP noop_request { }
I20260621 05:10:52.456981 11657 raft_consensus.cc:2866] T a1e067fcaf7749d39d931dbc49f28250 P eceee696a5ee4011b1264da66419eae9 [term 1 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted
I20260621 05:10:52.457017 11657 raft_consensus.cc:2258] T a1e067fcaf7749d39d931dbc49f28250 P eceee696a5ee4011b1264da66419eae9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 05:10:52.457114 11260 tablet_service.cc:1514] Processing DeleteTablet for tablet a1e067fcaf7749d39d931dbc49f28250 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 05:10:52 UTC) from {username='oryx', principal='oryx/127.3.131.190@KRBTEST.COM'} at 127.0.0.1:41098
I20260621 05:10:52.457203 11656 tablet_replica.cc:330] T a1e067fcaf7749d39d931dbc49f28250 P 6fa73262ba394487a5a54708f5de7075: stopping tablet replica
I20260621 05:10:52.457242 11656 raft_consensus.cc:2229] T a1e067fcaf7749d39d931dbc49f28250 P 6fa73262ba394487a5a54708f5de7075 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260621 05:10:52.457283 11656 raft_consensus.cc:2258] T a1e067fcaf7749d39d931dbc49f28250 P 6fa73262ba394487a5a54708f5de7075 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 05:10:52.457641 11122 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57322:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260621 05:10:52.457777 11122 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260621 05:10:52.458487 11657 ts_tablet_manager.cc:1905] T a1e067fcaf7749d39d931dbc49f28250 P eceee696a5ee4011b1264da66419eae9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 05:10:52.459312 11400 tablet_service.cc:1467] Processing CreateTablet for tablet 3df9a06f35b348d8811e55965c911eca (DEFAULT_TABLE table=test-table [id=70e79e14aca845cbade889c23097f1bb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 05:10:52.459432 11400 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3df9a06f35b348d8811e55965c911eca. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 05:10:52.460197 11260 tablet_service.cc:1467] Processing CreateTablet for tablet 3df9a06f35b348d8811e55965c911eca (DEFAULT_TABLE table=test-table [id=70e79e14aca845cbade889c23097f1bb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 05:10:52.460320 11260 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3df9a06f35b348d8811e55965c911eca. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 05:10:52.460505 11657 ts_tablet_manager.cc:1918] T a1e067fcaf7749d39d931dbc49f28250 P eceee696a5ee4011b1264da66419eae9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260621 05:10:52.460557 11657 log.cc:1196] T a1e067fcaf7749d39d931dbc49f28250 P eceee696a5ee4011b1264da66419eae9: Deleting WAL directory at /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782018616648578-3598-0/minicluster-data/ts-1/wal/wals/a1e067fcaf7749d39d931dbc49f28250
I20260621 05:10:52.460776 11657 ts_tablet_manager.cc:1939] T a1e067fcaf7749d39d931dbc49f28250 P eceee696a5ee4011b1264da66419eae9: Deleting consensus metadata
I20260621 05:10:52.461369 11634 tablet_bootstrap.cc:492] T 3df9a06f35b348d8811e55965c911eca P eceee696a5ee4011b1264da66419eae9: Bootstrap starting.
I20260621 05:10:52.461717 11634 tablet_bootstrap.cc:654] T 3df9a06f35b348d8811e55965c911eca P eceee696a5ee4011b1264da66419eae9: Neither blocks nor log segments found. Creating new log.
I20260621 05:10:52.461921 11637 tablet_bootstrap.cc:492] T 3df9a06f35b348d8811e55965c911eca P 6fa73262ba394487a5a54708f5de7075: Bootstrap starting.
I20260621 05:10:52.462257 11109 catalog_manager.cc:4841] TS eceee696a5ee4011b1264da66419eae9 (127.3.131.130:36509): tablet a1e067fcaf7749d39d931dbc49f28250 (table test-table [id=0916a34aeaf0400c80fc53514c8aa7d4]) successfully deleted
I20260621 05:10:52.462495 11637 tablet_bootstrap.cc:654] T 3df9a06f35b348d8811e55965c911eca P 6fa73262ba394487a5a54708f5de7075: Neither blocks nor log segments found. Creating new log.
I20260621 05:10:52.462658 11539 tablet_service.cc:1467] Processing CreateTablet for tablet 3df9a06f35b348d8811e55965c911eca (DEFAULT_TABLE table=test-table [id=70e79e14aca845cbade889c23097f1bb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 05:10:52.462782 11539 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3df9a06f35b348d8811e55965c911eca. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 05:10:52.462831 11658 ts_tablet_manager.cc:1905] T a1e067fcaf7749d39d931dbc49f28250 P 017e3652ab544ab08232482357ea8d48: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 05:10:52.463373 11637 tablet_bootstrap.cc:492] T 3df9a06f35b348d8811e55965c911eca P 6fa73262ba394487a5a54708f5de7075: No bootstrap required, opened a new log
I20260621 05:10:52.463421 11637 ts_tablet_manager.cc:1397] T 3df9a06f35b348d8811e55965c911eca P 6fa73262ba394487a5a54708f5de7075: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260621 05:10:52.463551 11637 raft_consensus.cc:348] T 3df9a06f35b348d8811e55965c911eca P 6fa73262ba394487a5a54708f5de7075 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fa73262ba394487a5a54708f5de7075" member_type: VOTER last_known_addr { host: "127.3.131.129" port: 38555 } } peers { permanent_uuid: "eceee696a5ee4011b1264da66419eae9" member_type: VOTER last_known_addr { host: "127.3.131.130" port: 36509 } } peers { permanent_uuid: "017e3652ab544ab08232482357ea8d48" member_type: VOTER last_known_addr { host: "127.3.131.131" port: 46349 } }
I20260621 05:10:52.463620 11637 raft_consensus.cc:374] T 3df9a06f35b348d8811e55965c911eca P 6fa73262ba394487a5a54708f5de7075 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 05:10:52.463641 11637 raft_consensus.cc:729] T 3df9a06f35b348d8811e55965c911eca P 6fa73262ba394487a5a54708f5de7075 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6fa73262ba394487a5a54708f5de7075, State: Initialized, Role: FOLLOWER
I20260621 05:10:52.463696 11637 consensus_queue.cc:260] T 3df9a06f35b348d8811e55965c911eca P 6fa73262ba394487a5a54708f5de7075 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fa73262ba394487a5a54708f5de7075" member_type: VOTER last_known_addr { host: "127.3.131.129" port: 38555 } } peers { permanent_uuid: "eceee696a5ee4011b1264da66419eae9" member_type: VOTER last_known_addr { host: "127.3.131.130" port: 36509 } } peers { permanent_uuid: "017e3652ab544ab08232482357ea8d48" member_type: VOTER last_known_addr { host: "127.3.131.131" port: 46349 } }
I20260621 05:10:52.463794 11637 ts_tablet_manager.cc:1428] T 3df9a06f35b348d8811e55965c911eca P 6fa73262ba394487a5a54708f5de7075: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260621 05:10:52.464198 11635 tablet_bootstrap.cc:492] T 3df9a06f35b348d8811e55965c911eca P 017e3652ab544ab08232482357ea8d48: Bootstrap starting.
I20260621 05:10:52.464617 11635 tablet_bootstrap.cc:654] T 3df9a06f35b348d8811e55965c911eca P 017e3652ab544ab08232482357ea8d48: Neither blocks nor log segments found. Creating new log.
I20260621 05:10:52.464956 11634 tablet_bootstrap.cc:492] T 3df9a06f35b348d8811e55965c911eca P eceee696a5ee4011b1264da66419eae9: No bootstrap required, opened a new log
I20260621 05:10:52.465027 11634 ts_tablet_manager.cc:1397] T 3df9a06f35b348d8811e55965c911eca P eceee696a5ee4011b1264da66419eae9: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20260621 05:10:52.465138 11634 raft_consensus.cc:348] T 3df9a06f35b348d8811e55965c911eca P eceee696a5ee4011b1264da66419eae9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fa73262ba394487a5a54708f5de7075" member_type: VOTER last_known_addr { host: "127.3.131.129" port: 38555 } } peers { permanent_uuid: "eceee696a5ee4011b1264da66419eae9" member_type: VOTER last_known_addr { host: "127.3.131.130" port: 36509 } } peers { permanent_uuid: "017e3652ab544ab08232482357ea8d48" member_type: VOTER last_known_addr { host: "127.3.131.131" port: 46349 } }
I20260621 05:10:52.465221 11634 raft_consensus.cc:374] T 3df9a06f35b348d8811e55965c911eca P eceee696a5ee4011b1264da66419eae9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 05:10:52.465243 11634 raft_consensus.cc:729] T 3df9a06f35b348d8811e55965c911eca P eceee696a5ee4011b1264da66419eae9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eceee696a5ee4011b1264da66419eae9, State: Initialized, Role: FOLLOWER
I20260621 05:10:52.465299 11635 tablet_bootstrap.cc:492] T 3df9a06f35b348d8811e55965c911eca P 017e3652ab544ab08232482357ea8d48: No bootstrap required, opened a new log
I20260621 05:10:52.465298 11634 consensus_queue.cc:260] T 3df9a06f35b348d8811e55965c911eca P eceee696a5ee4011b1264da66419eae9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fa73262ba394487a5a54708f5de7075" member_type: VOTER last_known_addr { host: "127.3.131.129" port: 38555 } } peers { permanent_uuid: "eceee696a5ee4011b1264da66419eae9" member_type: VOTER last_known_addr { host: "127.3.131.130" port: 36509 } } peers { permanent_uuid: "017e3652ab544ab08232482357ea8d48" member_type: VOTER last_known_addr { host: "127.3.131.131" port: 46349 } }
I20260621 05:10:52.465337 11635 ts_tablet_manager.cc:1397] T 3df9a06f35b348d8811e55965c911eca P 017e3652ab544ab08232482357ea8d48: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260621 05:10:52.465402 11634 ts_tablet_manager.cc:1428] T 3df9a06f35b348d8811e55965c911eca P eceee696a5ee4011b1264da66419eae9: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260621 05:10:52.465472 11635 raft_consensus.cc:348] T 3df9a06f35b348d8811e55965c911eca P 017e3652ab544ab08232482357ea8d48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fa73262ba394487a5a54708f5de7075" member_type: VOTER last_known_addr { host: "127.3.131.129" port: 38555 } } peers { permanent_uuid: "eceee696a5ee4011b1264da66419eae9" member_type: VOTER last_known_addr { host: "127.3.131.130" port: 36509 } } peers { permanent_uuid: "017e3652ab544ab08232482357ea8d48" member_type: VOTER last_known_addr { host: "127.3.131.131" port: 46349 } }
I20260621 05:10:52.465549 11635 raft_consensus.cc:374] T 3df9a06f35b348d8811e55965c911eca P 017e3652ab544ab08232482357ea8d48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 05:10:52.465570 11635 raft_consensus.cc:729] T 3df9a06f35b348d8811e55965c911eca P 017e3652ab544ab08232482357ea8d48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 017e3652ab544ab08232482357ea8d48, State: Initialized, Role: FOLLOWER
I20260621 05:10:52.465652 11635 consensus_queue.cc:260] T 3df9a06f35b348d8811e55965c911eca P 017e3652ab544ab08232482357ea8d48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fa73262ba394487a5a54708f5de7075" member_type: VOTER last_known_addr { host: "127.3.131.129" port: 38555 } } peers { permanent_uuid: "eceee696a5ee4011b1264da66419eae9" member_type: VOTER last_known_addr { host: "127.3.131.130" port: 36509 } } peers { permanent_uuid: "017e3652ab544ab08232482357ea8d48" member_type: VOTER last_known_addr { host: "127.3.131.131" port: 46349 } }
I20260621 05:10:52.465744 11635 ts_tablet_manager.cc:1428] T 3df9a06f35b348d8811e55965c911eca P 017e3652ab544ab08232482357ea8d48: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260621 05:10:52.466239 11658 ts_tablet_manager.cc:1918] T a1e067fcaf7749d39d931dbc49f28250 P 017e3652ab544ab08232482357ea8d48: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260621 05:10:52.466298 11658 log.cc:1196] T a1e067fcaf7749d39d931dbc49f28250 P 017e3652ab544ab08232482357ea8d48: Deleting WAL directory at /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782018616648578-3598-0/minicluster-data/ts-2/wal/wals/a1e067fcaf7749d39d931dbc49f28250
I20260621 05:10:52.466526 11658 ts_tablet_manager.cc:1939] T a1e067fcaf7749d39d931dbc49f28250 P 017e3652ab544ab08232482357ea8d48: Deleting consensus metadata
I20260621 05:10:52.466817 11108 catalog_manager.cc:4841] TS 017e3652ab544ab08232482357ea8d48 (127.3.131.131:46349): tablet a1e067fcaf7749d39d931dbc49f28250 (table test-table [id=0916a34aeaf0400c80fc53514c8aa7d4]) successfully deleted
I20260621 05:10:52.467159 11656 ts_tablet_manager.cc:1905] T a1e067fcaf7749d39d931dbc49f28250 P 6fa73262ba394487a5a54708f5de7075: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 05:10:52.468654 11656 ts_tablet_manager.cc:1918] T a1e067fcaf7749d39d931dbc49f28250 P 6fa73262ba394487a5a54708f5de7075: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260621 05:10:52.468720 11656 log.cc:1196] T a1e067fcaf7749d39d931dbc49f28250 P 6fa73262ba394487a5a54708f5de7075: Deleting WAL directory at /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782018616648578-3598-0/minicluster-data/ts-0/wal/wals/a1e067fcaf7749d39d931dbc49f28250
I20260621 05:10:52.468945 11656 ts_tablet_manager.cc:1939] T a1e067fcaf7749d39d931dbc49f28250 P 6fa73262ba394487a5a54708f5de7075: Deleting consensus metadata
I20260621 05:10:52.469489 11107 catalog_manager.cc:4841] TS 6fa73262ba394487a5a54708f5de7075 (127.3.131.129:38555): tablet a1e067fcaf7749d39d931dbc49f28250 (table test-table [id=0916a34aeaf0400c80fc53514c8aa7d4]) successfully deleted
I20260621 05:10:52.476071 11641 raft_consensus.cc:482] T 3df9a06f35b348d8811e55965c911eca P 6fa73262ba394487a5a54708f5de7075 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260621 05:10:52.476183 11641 raft_consensus.cc:504] T 3df9a06f35b348d8811e55965c911eca P 6fa73262ba394487a5a54708f5de7075 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fa73262ba394487a5a54708f5de7075" member_type: VOTER last_known_addr { host: "127.3.131.129" port: 38555 } } peers { permanent_uuid: "eceee696a5ee4011b1264da66419eae9" member_type: VOTER last_known_addr { host: "127.3.131.130" port: 36509 } } peers { permanent_uuid: "017e3652ab544ab08232482357ea8d48" member_type: VOTER last_known_addr { host: "127.3.131.131" port: 46349 } }
I20260621 05:10:52.476348 11641 leader_election.cc:288] T 3df9a06f35b348d8811e55965c911eca P 6fa73262ba394487a5a54708f5de7075 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers eceee696a5ee4011b1264da66419eae9 (127.3.131.130:36509), 017e3652ab544ab08232482357ea8d48 (127.3.131.131:46349)
I20260621 05:10:52.476605 11420 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3df9a06f35b348d8811e55965c911eca" candidate_uuid: "6fa73262ba394487a5a54708f5de7075" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eceee696a5ee4011b1264da66419eae9" is_pre_election: true
I20260621 05:10:52.476634 11560 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3df9a06f35b348d8811e55965c911eca" candidate_uuid: "6fa73262ba394487a5a54708f5de7075" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "017e3652ab544ab08232482357ea8d48" is_pre_election: true
I20260621 05:10:52.476727 11420 raft_consensus.cc:2454] T 3df9a06f35b348d8811e55965c911eca P eceee696a5ee4011b1264da66419eae9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6fa73262ba394487a5a54708f5de7075 in term 0.
I20260621 05:10:52.476727 11560 raft_consensus.cc:2454] T 3df9a06f35b348d8811e55965c911eca P 017e3652ab544ab08232482357ea8d48 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6fa73262ba394487a5a54708f5de7075 in term 0.
I20260621 05:10:52.476898 11213 leader_election.cc:302] T 3df9a06f35b348d8811e55965c911eca P 6fa73262ba394487a5a54708f5de7075 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6fa73262ba394487a5a54708f5de7075, eceee696a5ee4011b1264da66419eae9; no voters:
I20260621 05:10:52.477002 11641 raft_consensus.cc:2781] T 3df9a06f35b348d8811e55965c911eca P 6fa73262ba394487a5a54708f5de7075 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260621 05:10:52.477037 11641 raft_consensus.cc:482] T 3df9a06f35b348d8811e55965c911eca P 6fa73262ba394487a5a54708f5de7075 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260621 05:10:52.477067 11641 raft_consensus.cc:3037] T 3df9a06f35b348d8811e55965c911eca P 6fa73262ba394487a5a54708f5de7075 [term 0 FOLLOWER]: Advancing to term 1
I20260621 05:10:52.477751 11641 raft_consensus.cc:504] T 3df9a06f35b348d8811e55965c911eca P 6fa73262ba394487a5a54708f5de7075 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fa73262ba394487a5a54708f5de7075" member_type: VOTER last_known_addr { host: "127.3.131.129" port: 38555 } } peers { permanent_uuid: "eceee696a5ee4011b1264da66419eae9" member_type: VOTER last_known_addr { host: "127.3.131.130" port: 36509 } } peers { permanent_uuid: "017e3652ab544ab08232482357ea8d48" member_type: VOTER last_known_addr { host: "127.3.131.131" port: 46349 } }
I20260621 05:10:52.477902 11641 leader_election.cc:288] T 3df9a06f35b348d8811e55965c911eca P 6fa73262ba394487a5a54708f5de7075 [CANDIDATE]: Term 1 election: Requested vote from peers eceee696a5ee4011b1264da66419eae9 (127.3.131.130:36509), 017e3652ab544ab08232482357ea8d48 (127.3.131.131:46349)
I20260621 05:10:52.478034 11420 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3df9a06f35b348d8811e55965c911eca" candidate_uuid: "6fa73262ba394487a5a54708f5de7075" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eceee696a5ee4011b1264da66419eae9"
I20260621 05:10:52.478112 11420 raft_consensus.cc:3037] T 3df9a06f35b348d8811e55965c911eca P eceee696a5ee4011b1264da66419eae9 [term 0 FOLLOWER]: Advancing to term 1
I20260621 05:10:52.478127 11560 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3df9a06f35b348d8811e55965c911eca" candidate_uuid: "6fa73262ba394487a5a54708f5de7075" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "017e3652ab544ab08232482357ea8d48"
I20260621 05:10:52.478219 11560 raft_consensus.cc:3037] T 3df9a06f35b348d8811e55965c911eca P 017e3652ab544ab08232482357ea8d48 [term 0 FOLLOWER]: Advancing to term 1
I20260621 05:10:52.478818 11420 raft_consensus.cc:2454] T 3df9a06f35b348d8811e55965c911eca P eceee696a5ee4011b1264da66419eae9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6fa73262ba394487a5a54708f5de7075 in term 1.
I20260621 05:10:52.478998 11213 leader_election.cc:302] T 3df9a06f35b348d8811e55965c911eca P 6fa73262ba394487a5a54708f5de7075 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6fa73262ba394487a5a54708f5de7075, eceee696a5ee4011b1264da66419eae9; no voters:
I20260621 05:10:52.479055 11560 raft_consensus.cc:2454] T 3df9a06f35b348d8811e55965c911eca P 017e3652ab544ab08232482357ea8d48 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6fa73262ba394487a5a54708f5de7075 in term 1.
I20260621 05:10:52.479120 11641 raft_consensus.cc:2781] T 3df9a06f35b348d8811e55965c911eca P 6fa73262ba394487a5a54708f5de7075 [term 1 FOLLOWER]: Leader election won for term 1
I20260621 05:10:52.479192 11641 raft_consensus.cc:686] T 3df9a06f35b348d8811e55965c911eca P 6fa73262ba394487a5a54708f5de7075 [term 1 LEADER]: Becoming Leader. State: Replica: 6fa73262ba394487a5a54708f5de7075, State: Running, Role: LEADER
I20260621 05:10:52.479254 11641 consensus_queue.cc:237] T 3df9a06f35b348d8811e55965c911eca P 6fa73262ba394487a5a54708f5de7075 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fa73262ba394487a5a54708f5de7075" member_type: VOTER last_known_addr { host: "127.3.131.129" port: 38555 } } peers { permanent_uuid: "eceee696a5ee4011b1264da66419eae9" member_type: VOTER last_known_addr { host: "127.3.131.130" port: 36509 } } peers { permanent_uuid: "017e3652ab544ab08232482357ea8d48" member_type: VOTER last_known_addr { host: "127.3.131.131" port: 46349 } }
I20260621 05:10:52.479912 11122 catalog_manager.cc:5495] T 3df9a06f35b348d8811e55965c911eca P 6fa73262ba394487a5a54708f5de7075 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6fa73262ba394487a5a54708f5de7075 (127.3.131.129). New cstate: current_term: 1 leader_uuid: "6fa73262ba394487a5a54708f5de7075" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fa73262ba394487a5a54708f5de7075" member_type: VOTER last_known_addr { host: "127.3.131.129" port: 38555 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "eceee696a5ee4011b1264da66419eae9" member_type: VOTER last_known_addr { host: "127.3.131.130" port: 36509 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "017e3652ab544ab08232482357ea8d48" member_type: VOTER last_known_addr { host: "127.3.131.131" port: 46349 } health_report { overall_health: UNKNOWN } } }
I20260621 05:10:52.491420 11631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:10:52.489126 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41116 (local address 127.3.131.129:38555)
0621 05:10:52.489172 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0621 05:10:52.489174 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:10:52.489183 (+ 9us) server_negotiation.cc:407] Connection header received
0621 05:10:52.489228 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:10:52.489229 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:10:52.489239 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:10:52.489294 (+ 55us) server_negotiation.cc:226] Negotiated authn=TOKEN
0621 05:10:52.489395 (+ 101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:10:52.489891 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:10:52.490642 (+ 751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:10:52.490861 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:10:52.490946 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:10:52.491100 (+ 154us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0621 05:10:52.491135 (+ 35us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0621 05:10:52.491238 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0621 05:10:52.491302 (+ 64us) server_negotiation.cc:299] Negotiation successful
0621 05:10:52.491326 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260621 05:10:52.496443 11689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:10:52.493737 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.131.129:38555 (local address 127.3.131.129:50017)
0621 05:10:52.493826 (+ 89us) negotiation.cc:107] Waiting for socket to connect
0621 05:10:52.493840 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0621 05:10:52.493883 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:10:52.494002 (+ 119us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:10:52.494003 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:10:52.494004 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0621 05:10:52.494039 (+ 35us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:10:52.494040 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:10:52.494895 (+ 855us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:10:52.494896 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:10:52.495583 (+ 687us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:10:52.495585 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:10:52.496052 (+ 467us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:10:52.496053 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:10:52.496093 (+ 40us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:10:52.496099 (+ 6us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0621 05:10:52.496299 (+ 200us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0621 05:10:52.496300 (+ 1us) client_negotiation.cc:769] Sending connection context
0621 05:10:52.496335 (+ 35us) client_negotiation.cc:240] Negotiation successful
0621 05:10:52.496375 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":69}
I20260621 05:10:52.496454 11631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:10:52.493779 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.131.129:50017 (local address 127.3.131.129:38555)
0621 05:10:52.493835 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0621 05:10:52.493838 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0621 05:10:52.493871 (+ 33us) server_negotiation.cc:407] Connection header received
0621 05:10:52.493924 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:10:52.493925 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:10:52.493935 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:10:52.494002 (+ 67us) server_negotiation.cc:226] Negotiated authn=TOKEN
0621 05:10:52.494089 (+ 87us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:10:52.494805 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:10:52.495661 (+ 856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:10:52.495969 (+ 308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:10:52.496048 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:10:52.496208 (+ 160us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0621 05:10:52.496240 (+ 32us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0621 05:10:52.496292 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0621 05:10:52.496338 (+ 46us) server_negotiation.cc:299] Negotiation successful
0621 05:10:52.496359 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260621 05:10:52.498658 11560 raft_consensus.cc:1261] T 3df9a06f35b348d8811e55965c911eca P 017e3652ab544ab08232482357ea8d48 [term 1 FOLLOWER]: Refusing update from remote peer 6fa73262ba394487a5a54708f5de7075: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 05:10:52.498842 11420 raft_consensus.cc:1261] T 3df9a06f35b348d8811e55965c911eca P eceee696a5ee4011b1264da66419eae9 [term 1 FOLLOWER]: Refusing update from remote peer 6fa73262ba394487a5a54708f5de7075: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 05:10:52.498916 11641 consensus_queue.cc:1035] T 3df9a06f35b348d8811e55965c911eca P 6fa73262ba394487a5a54708f5de7075 [LEADER]: Connected to new peer: Peer: permanent_uuid: "017e3652ab544ab08232482357ea8d48" member_type: VOTER last_known_addr { host: "127.3.131.131" port: 46349 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 05:10:52.499027 11674 consensus_queue.cc:1035] T 3df9a06f35b348d8811e55965c911eca P 6fa73262ba394487a5a54708f5de7075 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eceee696a5ee4011b1264da66419eae9" member_type: VOTER last_known_addr { host: "127.3.131.130" port: 36509 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 05:10:52.514564 11122 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57322:
table { table_name: "test-table" } modify_external_catalogs: true
I20260621 05:10:52.514691 11122 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57322:
table { table_name: "test-table" } modify_external_catalogs: true
I20260621 05:10:52.515496 11122 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 65139302364d411cb10a6e712d783f40: Sending DeleteTablet for 3 replicas of tablet 3df9a06f35b348d8811e55965c911eca
I20260621 05:10:52.515880 11539 tablet_service.cc:1514] Processing DeleteTablet for tablet 3df9a06f35b348d8811e55965c911eca with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 05:10:52 UTC) from {username='oryx', principal='oryx/127.3.131.190@KRBTEST.COM'} at 127.0.0.1:38122
I20260621 05:10:52.515905 11400 tablet_service.cc:1514] Processing DeleteTablet for tablet 3df9a06f35b348d8811e55965c911eca with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 05:10:52 UTC) from {username='oryx', principal='oryx/127.3.131.190@KRBTEST.COM'} at 127.0.0.1:35264
I20260621 05:10:52.515980 11658 tablet_replica.cc:330] T 3df9a06f35b348d8811e55965c911eca P 017e3652ab544ab08232482357ea8d48: stopping tablet replica
I20260621 05:10:52.516017 11657 tablet_replica.cc:330] T 3df9a06f35b348d8811e55965c911eca P eceee696a5ee4011b1264da66419eae9: stopping tablet replica
I20260621 05:10:52.516022 11658 raft_consensus.cc:2229] T 3df9a06f35b348d8811e55965c911eca P 017e3652ab544ab08232482357ea8d48 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260621 05:10:52.516062 11658 raft_consensus.cc:2258] T 3df9a06f35b348d8811e55965c911eca P 017e3652ab544ab08232482357ea8d48 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 05:10:52.516134 11657 raft_consensus.cc:2229] T 3df9a06f35b348d8811e55965c911eca P eceee696a5ee4011b1264da66419eae9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260621 05:10:52.516181 11657 raft_consensus.cc:2258] T 3df9a06f35b348d8811e55965c911eca P eceee696a5ee4011b1264da66419eae9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 05:10:52.516273 11259 tablet_service.cc:1514] Processing DeleteTablet for tablet 3df9a06f35b348d8811e55965c911eca with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 05:10:52 UTC) from {username='oryx', principal='oryx/127.3.131.190@KRBTEST.COM'} at 127.0.0.1:41098
I20260621 05:10:52.516361 11656 tablet_replica.cc:330] T 3df9a06f35b348d8811e55965c911eca P 6fa73262ba394487a5a54708f5de7075: stopping tablet replica
I20260621 05:10:52.516403 11656 raft_consensus.cc:2229] T 3df9a06f35b348d8811e55965c911eca P 6fa73262ba394487a5a54708f5de7075 [term 1 LEADER]: Raft consensus shutting down.
I20260621 05:10:52.516503 11656 raft_consensus.cc:2258] T 3df9a06f35b348d8811e55965c911eca P 6fa73262ba394487a5a54708f5de7075 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 05:10:52.516973 11658 ts_tablet_manager.cc:1905] T 3df9a06f35b348d8811e55965c911eca P 017e3652ab544ab08232482357ea8d48: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 05:10:52.516997 11656 ts_tablet_manager.cc:1905] T 3df9a06f35b348d8811e55965c911eca P 6fa73262ba394487a5a54708f5de7075: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 05:10:52.518795 11658 ts_tablet_manager.cc:1918] T 3df9a06f35b348d8811e55965c911eca P 017e3652ab544ab08232482357ea8d48: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260621 05:10:52.518884 11658 log.cc:1196] T 3df9a06f35b348d8811e55965c911eca P 017e3652ab544ab08232482357ea8d48: Deleting WAL directory at /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782018616648578-3598-0/minicluster-data/ts-2/wal/wals/3df9a06f35b348d8811e55965c911eca
I20260621 05:10:52.518924 11657 ts_tablet_manager.cc:1905] T 3df9a06f35b348d8811e55965c911eca P eceee696a5ee4011b1264da66419eae9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 05:10:52.519115 11658 ts_tablet_manager.cc:1939] T 3df9a06f35b348d8811e55965c911eca P 017e3652ab544ab08232482357ea8d48: Deleting consensus metadata
I20260621 05:10:52.519433 11656 ts_tablet_manager.cc:1918] T 3df9a06f35b348d8811e55965c911eca P 6fa73262ba394487a5a54708f5de7075: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260621 05:10:52.519493 11656 log.cc:1196] T 3df9a06f35b348d8811e55965c911eca P 6fa73262ba394487a5a54708f5de7075: Deleting WAL directory at /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782018616648578-3598-0/minicluster-data/ts-0/wal/wals/3df9a06f35b348d8811e55965c911eca
I20260621 05:10:52.519558 3598 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWZOx2_/build/release/bin/kudu with pid 11195
I20260621 05:10:52.519711 11656 ts_tablet_manager.cc:1939] T 3df9a06f35b348d8811e55965c911eca P 6fa73262ba394487a5a54708f5de7075: Deleting consensus metadata
I20260621 05:10:52.520671 11108 catalog_manager.cc:4841] TS 017e3652ab544ab08232482357ea8d48 (127.3.131.131:46349): tablet 3df9a06f35b348d8811e55965c911eca (table test-table [id=70e79e14aca845cbade889c23097f1bb]) successfully deleted
I20260621 05:10:52.522116 11657 ts_tablet_manager.cc:1918] T 3df9a06f35b348d8811e55965c911eca P eceee696a5ee4011b1264da66419eae9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260621 05:10:52.522190 11657 log.cc:1196] T 3df9a06f35b348d8811e55965c911eca P eceee696a5ee4011b1264da66419eae9: Deleting WAL directory at /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782018616648578-3598-0/minicluster-data/ts-1/wal/wals/3df9a06f35b348d8811e55965c911eca
I20260621 05:10:52.522428 11657 ts_tablet_manager.cc:1939] T 3df9a06f35b348d8811e55965c911eca P eceee696a5ee4011b1264da66419eae9: Deleting consensus metadata
I20260621 05:10:52.522814 11109 catalog_manager.cc:4841] TS eceee696a5ee4011b1264da66419eae9 (127.3.131.130:36509): tablet 3df9a06f35b348d8811e55965c911eca (table test-table [id=70e79e14aca845cbade889c23097f1bb]) successfully deleted
W20260621 05:10:52.528887 11107 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.3.131.129:38555 (error 108)
I20260621 05:10:52.529539 11723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:10:52.529327 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.131.129:38555 (local address 127.0.0.1:41130)
0621 05:10:52.529444 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0621 05:10:52.529493 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.131.129:38555: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99,"thread_start_us":62,"threads_started":1}
I20260621 05:10:52.529515 3598 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWZOx2_/build/release/bin/kudu with pid 11335
W20260621 05:10:52.531607 11107 catalog_manager.cc:4584] TS 6fa73262ba394487a5a54708f5de7075 (127.3.131.129:38555): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 3df9a06f35b348d8811e55965c911eca: Network error: Client connection negotiation failed: client connection to 127.3.131.129:38555: connect: Connection refused (error 111)
I20260621 05:10:52.538442 3598 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWZOx2_/build/release/bin/kudu with pid 11475
I20260621 05:10:52.545966 3598 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWZOx2_/build/release/bin/kudu with pid 11091
2026-06-21T05:10:52Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (2614 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1782018616648578-3598-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 21 05:10:52 dist-test-slave-vrq1 krb5kdc[11726](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 21 05:10:52 dist-test-slave-vrq1 krb5kdc[11726](info): set up 2 sockets
Jun 21 05:10:52 dist-test-slave-vrq1 krb5kdc[11726](info): commencing operation
krb5kdc: starting...
W20260621 05:10:53.608357 3598 mini_kdc.cc:121] Time spent starting KDC: real 1.039s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1782018616648578-3598-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1782018616648578-3598-0/krb5kdc/test-user.keytab.
Jun 21 05:10:53 dist-test-slave-vrq1 krb5kdc[11726](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782018653, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-21T05:10:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-21T05:10:53Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.131.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.131.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.131.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1782018616648578-3598-0/krb5kdc/kudu_127.3.131.190.keytab.
Entry for principal kudu/127.3.131.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1782018616648578-3598-0/krb5kdc/kudu_127.3.131.190.keytab.
I20260621 05:10:53.740114 3598 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWZOx2_/build/release/bin/kudu
/tmp/dist-test-taskWZOx2_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1782018616648578-3598-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1782018616648578-3598-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1782018616648578-3598-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1782018616648578-3598-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.131.190:39057
--webserver_interface=127.3.131.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1782018616648578-3598-0/krb5kdc/kudu_127.3.131.190.keytab
--principal=kudu/127.3.131.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.131.148:34655
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.131.190:39057
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1782018616648578-3598-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1782018616648578-3598-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1782018616648578-3598-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 05:10:53.830444 11739 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 05:10:53.830698 11739 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 05:10:53.830725 11739 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 05:10:53.832072 11739 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 05:10:53.832121 11739 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 05:10:53.832136 11739 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 05:10:53.832149 11739 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 05:10:53.832161 11739 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 05:10:53.833814 11739 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.131.148:34655
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1782018616648578-3598-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1782018616648578-3598-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.131.190:39057
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1782018616648578-3598-0/krb5kdc/kudu_127.3.131.190.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.131.190:39057
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1782018616648578-3598-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.131.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11739
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1782018616648578-3598-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 21 Jun 2026 04:16:16 UTC on ccycloud.quasar-xycaha.root.comops.site
build id 4457881
I20260621 05:10:53.834115 11739 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 05:10:53.834374 11739 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 05:10:53.839761 11747 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:10:53.839761 11744 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:10:53.839769 11745 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:10:53.839792 11739 server_base.cc:1034] running on GCE node
I20260621 05:10:53.840328 11739 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 05:10:53.840636 11739 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 05:10:53.841781 11739 hybrid_clock.cc:648] HybridClock initialized: now 1782018653841773 us; error 31 us; skew 500 ppm
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-06-21T05:10:53Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1300 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782018616648578-3598-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 21 05:10:53 dist-test-slave-vrq1 krb5kdc[11752](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 21 05:10:53 dist-test-slave-vrq1 krb5kdc[11752](info): set up 2 sockets
Jun 21 05:10:53 dist-test-slave-vrq1 krb5kdc[11752](info): commencing operation
krb5kdc: starting...
W20260621 05:10:54.928232 3598 mini_kdc.cc:121] Time spent starting KDC: real 1.059s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782018616648578-3598-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782018616648578-3598-0/krb5kdc/test-user.keytab.
Jun 21 05:10:55 dist-test-slave-vrq1 krb5kdc[11752](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782018655, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-21T05:10:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-21T05:10:55Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.131.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.131.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.131.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782018616648578-3598-0/krb5kdc/kudu_127.3.131.190.keytab.
Entry for principal kudu/127.3.131.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782018616648578-3598-0/krb5kdc/kudu_127.3.131.190.keytab.
I20260621 05:10:55.056468 3598 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWZOx2_/build/release/bin/kudu
/tmp/dist-test-taskWZOx2_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782018616648578-3598-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782018616648578-3598-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782018616648578-3598-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782018616648578-3598-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.131.190:37467
--webserver_interface=127.3.131.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782018616648578-3598-0/krb5kdc/kudu_127.3.131.190.keytab
--principal=kudu/127.3.131.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.131.148:42409
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.131.190:37467
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782018616648578-3598-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782018616648578-3598-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782018616648578-3598-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 05:10:55.140914 11765 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 05:10:55.141175 11765 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 05:10:55.141216 11765 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 05:10:55.142458 11765 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 05:10:55.142531 11765 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 05:10:55.142555 11765 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 05:10:55.142580 11765 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 05:10:55.142608 11765 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 05:10:55.144223 11765 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.131.148:42409
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782018616648578-3598-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782018616648578-3598-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.131.190:37467
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782018616648578-3598-0/krb5kdc/kudu_127.3.131.190.keytab
--principal=kudu/127.3.131.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.131.190:37467
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782018616648578-3598-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.131.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11765
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782018616648578-3598-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 21 Jun 2026 04:16:16 UTC on ccycloud.quasar-xycaha.root.comops.site
build id 4457881
I20260621 05:10:55.144526 11765 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 05:10:55.144788 11765 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 05:10:55.149988 11771 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:10:55.150060 11770 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:10:55.150197 11765 server_base.cc:1034] running on GCE node
W20260621 05:10:55.149993 11773 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:10:55.150557 11765 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 05:10:55.150878 11765 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 05:10:55.152113 11765 hybrid_clock.cc:648] HybridClock initialized: now 1782018655152075 us; error 55 us; skew 500 ppm
Jun 21 05:10:55 dist-test-slave-vrq1 krb5kdc[11752](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782018655, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.131.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 05:10:55.154670 11765 init.cc:377] Logged in from keytab as kudu/127.3.131.190@KRBTEST.COM (short username kudu)
I20260621 05:10:55.155310 11765 webserver.cc:460] Webserver started at http://127.3.131.190:36615/ using document root <none> and password file <none>
I20260621 05:10:55.155577 11765 fs_manager.cc:359] Metadata directory not provided
I20260621 05:10:55.155650 11765 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 05:10:55.155817 11765 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 05:10:55.157006 11765 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782018616648578-3598-0/minicluster-data/master-0/data/instance:
uuid: "5ab2baa2ec874efdbbae3f8fc138ff73"
format_stamp: "Formatted at 2026-06-21 05:10:55 on dist-test-slave-vrq1"
server_key: "8893e8723530060035981ac73d9c20e4"
server_key_iv: "a4932200db912dc1037f9e7f0531a8ea"
server_key_version: "encryptionkey@0"
I20260621 05:10:55.157408 11765 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782018616648578-3598-0/minicluster-data/master-0/wal/instance:
uuid: "5ab2baa2ec874efdbbae3f8fc138ff73"
format_stamp: "Formatted at 2026-06-21 05:10:55 on dist-test-slave-vrq1"
server_key: "8893e8723530060035981ac73d9c20e4"
server_key_iv: "a4932200db912dc1037f9e7f0531a8ea"
server_key_version: "encryptionkey@0"
I20260621 05:10:55.158946 11765 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260621 05:10:55.160025 11780 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 05:10:55.160279 11765 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260621 05:10:55.160401 11765 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782018616648578-3598-0/minicluster-data/master-0/data,/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782018616648578-3598-0/minicluster-data/master-0/wal
uuid: "5ab2baa2ec874efdbbae3f8fc138ff73"
format_stamp: "Formatted at 2026-06-21 05:10:55 on dist-test-slave-vrq1"
server_key: "8893e8723530060035981ac73d9c20e4"
server_key_iv: "a4932200db912dc1037f9e7f0531a8ea"
server_key_version: "encryptionkey@0"
I20260621 05:10:55.160490 11765 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782018616648578-3598-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782018616648578-3598-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782018616648578-3598-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 05:10:55.170744 11765 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 05:10:55.173122 11765 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 05:10:55.173352 11765 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 05:10:55.177551 11765 rpc_server.cc:307] RPC server started. Bound to: 127.3.131.190:37467
I20260621 05:10:55.177587 11832 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.131.190:37467 every 8 connection(s)
I20260621 05:10:55.177964 11765 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782018616648578-3598-0/minicluster-data/master-0/data/info.pb
I20260621 05:10:55.179018 11833 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 05:10:55.181576 11833 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5ab2baa2ec874efdbbae3f8fc138ff73: Bootstrap starting.
I20260621 05:10:55.182199 11833 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5ab2baa2ec874efdbbae3f8fc138ff73: Neither blocks nor log segments found. Creating new log.
I20260621 05:10:55.182194 3598 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWZOx2_/build/release/bin/kudu as pid 11765
I20260621 05:10:55.182310 3598 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782018616648578-3598-0/minicluster-data/master-0/wal/instance
I20260621 05:10:55.182507 3598 external_mini_cluster.cc:1428] Setting key a2b9c2581f1a2c2a1fb230ed17b60ace
I20260621 05:10:55.182545 11833 log.cc:824] T 00000000000000000000000000000000 P 5ab2baa2ec874efdbbae3f8fc138ff73: Log is configured to *not* fsync() on all Append() calls
I20260621 05:10:55.183630 11833 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5ab2baa2ec874efdbbae3f8fc138ff73: No bootstrap required, opened a new log
I20260621 05:10:55.186152 11833 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5ab2baa2ec874efdbbae3f8fc138ff73 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ab2baa2ec874efdbbae3f8fc138ff73" member_type: VOTER last_known_addr { host: "127.3.131.190" port: 37467 } }
I20260621 05:10:55.186333 11833 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5ab2baa2ec874efdbbae3f8fc138ff73 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 05:10:55.186365 11833 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5ab2baa2ec874efdbbae3f8fc138ff73 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5ab2baa2ec874efdbbae3f8fc138ff73, State: Initialized, Role: FOLLOWER
I20260621 05:10:55.186496 11833 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5ab2baa2ec874efdbbae3f8fc138ff73 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ab2baa2ec874efdbbae3f8fc138ff73" member_type: VOTER last_known_addr { host: "127.3.131.190" port: 37467 } }
I20260621 05:10:55.186585 11833 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5ab2baa2ec874efdbbae3f8fc138ff73 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260621 05:10:55.186630 11833 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5ab2baa2ec874efdbbae3f8fc138ff73 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260621 05:10:55.186681 11833 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5ab2baa2ec874efdbbae3f8fc138ff73 [term 0 FOLLOWER]: Advancing to term 1
I20260621 05:10:55.187330 11833 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5ab2baa2ec874efdbbae3f8fc138ff73 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ab2baa2ec874efdbbae3f8fc138ff73" member_type: VOTER last_known_addr { host: "127.3.131.190" port: 37467 } }
I20260621 05:10:55.187455 11833 leader_election.cc:302] T 00000000000000000000000000000000 P 5ab2baa2ec874efdbbae3f8fc138ff73 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5ab2baa2ec874efdbbae3f8fc138ff73; no voters:
I20260621 05:10:55.187641 11833 leader_election.cc:288] T 00000000000000000000000000000000 P 5ab2baa2ec874efdbbae3f8fc138ff73 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260621 05:10:55.187767 11838 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5ab2baa2ec874efdbbae3f8fc138ff73 [term 1 FOLLOWER]: Leader election won for term 1
I20260621 05:10:55.187928 11833 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5ab2baa2ec874efdbbae3f8fc138ff73 [sys.catalog]: configured and running, proceeding with master startup.
I20260621 05:10:55.187968 11838 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5ab2baa2ec874efdbbae3f8fc138ff73 [term 1 LEADER]: Becoming Leader. State: Replica: 5ab2baa2ec874efdbbae3f8fc138ff73, State: Running, Role: LEADER
I20260621 05:10:55.188220 11838 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5ab2baa2ec874efdbbae3f8fc138ff73 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ab2baa2ec874efdbbae3f8fc138ff73" member_type: VOTER last_known_addr { host: "127.3.131.190" port: 37467 } }
I20260621 05:10:55.188870 11840 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5ab2baa2ec874efdbbae3f8fc138ff73 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5ab2baa2ec874efdbbae3f8fc138ff73" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ab2baa2ec874efdbbae3f8fc138ff73" member_type: VOTER last_known_addr { host: "127.3.131.190" port: 37467 } } }
I20260621 05:10:55.188951 11840 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5ab2baa2ec874efdbbae3f8fc138ff73 [sys.catalog]: This master's current role is: LEADER
I20260621 05:10:55.189193 11839 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5ab2baa2ec874efdbbae3f8fc138ff73 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5ab2baa2ec874efdbbae3f8fc138ff73. Latest consensus state: current_term: 1 leader_uuid: "5ab2baa2ec874efdbbae3f8fc138ff73" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ab2baa2ec874efdbbae3f8fc138ff73" member_type: VOTER last_known_addr { host: "127.3.131.190" port: 37467 } } }
I20260621 05:10:55.189316 11839 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5ab2baa2ec874efdbbae3f8fc138ff73 [sys.catalog]: This master's current role is: LEADER
I20260621 05:10:55.189488 11848 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260621 05:10:55.189949 11848 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Jun 21 05:10:55 dist-test-slave-vrq1 krb5kdc[11752](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782018655, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.131.190@KRBTEST.COM
I20260621 05:10:55.191577 11848 catalog_manager.cc:1347] Generated new cluster ID: 6dd7dd2b0d704f138f788517f9796ba6
I20260621 05:10:55.191630 11848 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260621 05:10:55.195501 11837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:10:55.183160 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53238 (local address 127.3.131.190:37467)
0621 05:10:55.183353 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0621 05:10:55.183362 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0621 05:10:55.183398 (+ 36us) server_negotiation.cc:407] Connection header received
0621 05:10:55.183998 (+ 600us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:10:55.184007 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:10:55.184083 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:10:55.184257 (+ 174us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:10:55.184909 (+ 652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:10:55.186414 (+ 1505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:10:55.187391 (+ 977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:10:55.188039 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:10:55.188211 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:10:55.191415 (+ 3204us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:10:55.191419 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:10:55.191419 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:10:55.191433 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:10:55.193398 (+ 1965us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:10:55.193811 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:10:55.193815 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:10:55.193815 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:10:55.193911 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:10:55.194089 (+ 178us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:10:55.194089 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:10:55.194090 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:10:55.194372 (+ 282us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:10:55.194520 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0621 05:10:55.194965 (+ 445us) server_negotiation.cc:299] Negotiation successful
0621 05:10:55.195096 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":119,"thread_start_us":86,"threads_started":1}
I20260621 05:10:55.224465 11848 catalog_manager.cc:1370] Generated new certificate authority record
I20260621 05:10:55.225057 11848 catalog_manager.cc:1504] Loading token signing keys...
I20260621 05:10:55.230721 11848 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5ab2baa2ec874efdbbae3f8fc138ff73: Generated new TSK 0
I20260621 05:10:55.230922 11848 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.3.131.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.131.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.131.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782018616648578-3598-0/krb5kdc/kudu_127.3.131.129.keytab.
Entry for principal kudu/127.3.131.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782018616648578-3598-0/krb5kdc/kudu_127.3.131.129.keytab.
I20260621 05:10:55.262401 3598 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWZOx2_/build/release/bin/kudu
/tmp/dist-test-taskWZOx2_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782018616648578-3598-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782018616648578-3598-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782018616648578-3598-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782018616648578-3598-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.131.129:0
--local_ip_for_outbound_sockets=127.3.131.129
--webserver_interface=127.3.131.129
--webserver_port=0
--tserver_master_addrs=127.3.131.190:37467
--keytab_file=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782018616648578-3598-0/krb5kdc/kudu_127.3.131.129.keytab
--principal=kudu/127.3.131.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.131.148:42409
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782018616648578-3598-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782018616648578-3598-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782018616648578-3598-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 05:10:55.378932 11859 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 05:10:55.379208 11859 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 05:10:55.379238 11859 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 05:10:55.380504 11859 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 05:10:55.380553 11859 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 05:10:55.380582 11859 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.131.129
I20260621 05:10:55.382133 11859 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.131.148:42409
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782018616648578-3598-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782018616648578-3598-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782018616648578-3598-0/krb5kdc/kudu_127.3.131.129.keytab
--principal=kudu/127.3.131.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.131.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782018616648578-3598-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.131.129
--webserver_port=0
--tserver_master_addrs=127.3.131.190:37467
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11859
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.131.129
--log_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782018616648578-3598-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 21 Jun 2026 04:16:16 UTC on ccycloud.quasar-xycaha.root.comops.site
build id 4457881
I20260621 05:10:55.382362 11859 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 05:10:55.382563 11859 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 05:10:55.387940 11867 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:10:55.387933 11865 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:10:55.387971 11864 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:10:55.388067 11859 server_base.cc:1034] running on GCE node
I20260621 05:10:55.388382 11859 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 05:10:55.388643 11859 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 05:10:55.389813 11859 hybrid_clock.cc:648] HybridClock initialized: now 1782018655389744 us; error 97 us; skew 500 ppm
Jun 21 05:10:55 dist-test-slave-vrq1 krb5kdc[11752](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782018655, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.131.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 05:10:55.392598 11859 init.cc:377] Logged in from keytab as kudu/127.3.131.129@KRBTEST.COM (short username kudu)
I20260621 05:10:55.393126 11859 webserver.cc:460] Webserver started at http://127.3.131.129:35113/ using document root <none> and password file <none>
I20260621 05:10:55.393461 11859 fs_manager.cc:359] Metadata directory not provided
I20260621 05:10:55.393527 11859 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 05:10:55.393700 11859 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 05:10:55.394984 11859 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782018616648578-3598-0/minicluster-data/ts-0/data/instance:
uuid: "4623cc8171b7486bb1b0d5bdcd71fdd5"
format_stamp: "Formatted at 2026-06-21 05:10:55 on dist-test-slave-vrq1"
server_key: "8f18b0ad387973f1fc97692097d7c285"
server_key_iv: "5db7fb9a2f33e7d1603f8fa357d41ebb"
server_key_version: "encryptionkey@0"
I20260621 05:10:55.395673 11859 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782018616648578-3598-0/minicluster-data/ts-0/wal/instance:
uuid: "4623cc8171b7486bb1b0d5bdcd71fdd5"
format_stamp: "Formatted at 2026-06-21 05:10:55 on dist-test-slave-vrq1"
server_key: "8f18b0ad387973f1fc97692097d7c285"
server_key_iv: "5db7fb9a2f33e7d1603f8fa357d41ebb"
server_key_version: "encryptionkey@0"
I20260621 05:10:55.397272 11859 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260621 05:10:55.398268 11874 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 05:10:55.398550 11859 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260621 05:10:55.398694 11859 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782018616648578-3598-0/minicluster-data/ts-0/data,/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782018616648578-3598-0/minicluster-data/ts-0/wal
uuid: "4623cc8171b7486bb1b0d5bdcd71fdd5"
format_stamp: "Formatted at 2026-06-21 05:10:55 on dist-test-slave-vrq1"
server_key: "8f18b0ad387973f1fc97692097d7c285"
server_key_iv: "5db7fb9a2f33e7d1603f8fa357d41ebb"
server_key_version: "encryptionkey@0"
I20260621 05:10:55.398779 11859 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782018616648578-3598-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782018616648578-3598-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782018616648578-3598-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 05:10:55.410482 11859 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 05:10:55.412981 11859 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 05:10:55.413141 11859 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 05:10:55.413415 11859 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 05:10:55.413766 11859 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 05:10:55.413825 11859 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 05:10:55.413861 11859 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 05:10:55.413887 11859 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 05:10:55.420207 11859 rpc_server.cc:307] RPC server started. Bound to: 127.3.131.129:35715
I20260621 05:10:55.420250 11987 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.131.129:35715 every 8 connection(s)
I20260621 05:10:55.420588 11859 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782018616648578-3598-0/minicluster-data/ts-0/data/info.pb
Jun 21 05:10:55 dist-test-slave-vrq1 krb5kdc[11752](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782018655, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.131.129@KRBTEST.COM for kudu/127.3.131.190@KRBTEST.COM
I20260621 05:10:55.430029 11837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:10:55.421402 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.131.129:54121 (local address 127.3.131.190:37467)
0621 05:10:55.421497 (+ 95us) server_negotiation.cc:206] Beginning negotiation
0621 05:10:55.421500 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0621 05:10:55.421696 (+ 196us) server_negotiation.cc:407] Connection header received
0621 05:10:55.422162 (+ 466us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:10:55.422164 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:10:55.422174 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:10:55.422232 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:10:55.423154 (+ 922us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:10:55.423772 (+ 618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:10:55.424837 (+ 1065us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:10:55.425067 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:10:55.425140 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:10:55.427400 (+ 2260us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:10:55.427401 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:10:55.427401 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:10:55.427409 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:10:55.428769 (+ 1360us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:10:55.429154 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:10:55.429155 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:10:55.429156 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:10:55.429214 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:10:55.429437 (+ 223us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:10:55.429438 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:10:55.429438 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:10:55.429550 (+ 112us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:10:55.429601 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0621 05:10:55.429882 (+ 281us) server_negotiation.cc:299] Negotiation successful
0621 05:10:55.429961 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260621 05:10:55.430111 11990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:10:55.421381 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.131.190:37467 (local address 127.3.131.129:54121)
0621 05:10:55.421569 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0621 05:10:55.421612 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0621 05:10:55.422057 (+ 445us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:10:55.422288 (+ 231us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:10:55.422291 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:10:55.422854 (+ 563us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:10:55.423048 (+ 194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:10:55.423053 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:10:55.423866 (+ 813us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:10:55.423867 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:10:55.424726 (+ 859us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:10:55.424728 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:10:55.425164 (+ 436us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:10:55.425164 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:10:55.425279 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:10:55.425868 (+ 589us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 05:10:55.425876 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:10:55.427297 (+ 1421us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 05:10:55.428847 (+ 1550us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:10:55.428852 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:10:55.428854 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:10:55.429071 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:10:55.429283 (+ 212us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:10:55.429283 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:10:55.429284 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:10:55.429381 (+ 97us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:10:55.429624 (+ 243us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 05:10:55.429628 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 05:10:55.429742 (+ 114us) client_negotiation.cc:769] Sending connection context
0621 05:10:55.429827 (+ 85us) client_negotiation.cc:240] Negotiation successful
0621 05:10:55.429949 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":109,"threads_started":1}
I20260621 05:10:55.430344 3598 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWZOx2_/build/release/bin/kudu as pid 11859
I20260621 05:10:55.430433 3598 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782018616648578-3598-0/minicluster-data/ts-0/wal/instance
I20260621 05:10:55.430596 11988 heartbeater.cc:347] Connected to a master server at 127.3.131.190:37467
I20260621 05:10:55.430645 3598 external_mini_cluster.cc:1428] Setting key a5329a87125359dbd6bd430abdfde8af
I20260621 05:10:55.430696 11988 heartbeater.cc:464] Registering TS with master...
I20260621 05:10:55.430900 11988 heartbeater.cc:511] Master 127.3.131.190:37467 requested a full tablet report, sending...
I20260621 05:10:55.431445 11796 ts_manager.cc:194] Registered new tserver with Master: 4623cc8171b7486bb1b0d5bdcd71fdd5 (127.3.131.129:35715)
I20260621 05:10:55.432056 11796 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.131.129@KRBTEST.COM'} at 127.3.131.129:54121
WARNING: no policy specified for kudu/127.3.131.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.131.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.131.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782018616648578-3598-0/krb5kdc/kudu_127.3.131.130.keytab.
Entry for principal kudu/127.3.131.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782018616648578-3598-0/krb5kdc/kudu_127.3.131.130.keytab.
I20260621 05:10:55.459476 3598 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWZOx2_/build/release/bin/kudu
/tmp/dist-test-taskWZOx2_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782018616648578-3598-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782018616648578-3598-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782018616648578-3598-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782018616648578-3598-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.131.130:0
--local_ip_for_outbound_sockets=127.3.131.130
--webserver_interface=127.3.131.130
--webserver_port=0
--tserver_master_addrs=127.3.131.190:37467
--keytab_file=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782018616648578-3598-0/krb5kdc/kudu_127.3.131.130.keytab
--principal=kudu/127.3.131.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.131.148:42409
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782018616648578-3598-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782018616648578-3598-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782018616648578-3598-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 05:10:55.544503 11993 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 05:10:55.544734 11993 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 05:10:55.544760 11993 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 05:10:55.546042 11993 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 05:10:55.546096 11993 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 05:10:55.546125 11993 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.131.130
I20260621 05:10:55.547647 11993 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.131.148:42409
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782018616648578-3598-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782018616648578-3598-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782018616648578-3598-0/krb5kdc/kudu_127.3.131.130.keytab
--principal=kudu/127.3.131.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.131.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782018616648578-3598-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.131.130
--webserver_port=0
--tserver_master_addrs=127.3.131.190:37467
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11993
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.131.130
--log_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782018616648578-3598-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 21 Jun 2026 04:16:16 UTC on ccycloud.quasar-xycaha.root.comops.site
build id 4457881
I20260621 05:10:55.547910 11993 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 05:10:55.548157 11993 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 05:10:55.553594 11998 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:10:55.553575 12001 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:10:55.553578 11999 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:10:55.553642 11993 server_base.cc:1034] running on GCE node
I20260621 05:10:55.554008 11993 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 05:10:55.554275 11993 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 05:10:55.555442 11993 hybrid_clock.cc:648] HybridClock initialized: now 1782018655555418 us; error 43 us; skew 500 ppm
Jun 21 05:10:55 dist-test-slave-vrq1 krb5kdc[11752](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782018655, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.131.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 05:10:55.558543 11993 init.cc:377] Logged in from keytab as kudu/127.3.131.130@KRBTEST.COM (short username kudu)
I20260621 05:10:55.559020 11993 webserver.cc:460] Webserver started at http://127.3.131.130:40481/ using document root <none> and password file <none>
I20260621 05:10:55.559300 11993 fs_manager.cc:359] Metadata directory not provided
I20260621 05:10:55.559350 11993 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 05:10:55.559497 11993 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 05:10:55.560562 11993 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782018616648578-3598-0/minicluster-data/ts-1/data/instance:
uuid: "83e89bf9905742ec80633a26819621ae"
format_stamp: "Formatted at 2026-06-21 05:10:55 on dist-test-slave-vrq1"
server_key: "1b7599748098fde9410a4cd4d8d7a792"
server_key_iv: "b2f3a17f96be00c8c47a98b1e8e88d44"
server_key_version: "encryptionkey@0"
I20260621 05:10:55.560937 11993 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782018616648578-3598-0/minicluster-data/ts-1/wal/instance:
uuid: "83e89bf9905742ec80633a26819621ae"
format_stamp: "Formatted at 2026-06-21 05:10:55 on dist-test-slave-vrq1"
server_key: "1b7599748098fde9410a4cd4d8d7a792"
server_key_iv: "b2f3a17f96be00c8c47a98b1e8e88d44"
server_key_version: "encryptionkey@0"
I20260621 05:10:55.562376 11993 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260621 05:10:55.563309 12008 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 05:10:55.563515 11993 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260621 05:10:55.563645 11993 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782018616648578-3598-0/minicluster-data/ts-1/data,/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782018616648578-3598-0/minicluster-data/ts-1/wal
uuid: "83e89bf9905742ec80633a26819621ae"
format_stamp: "Formatted at 2026-06-21 05:10:55 on dist-test-slave-vrq1"
server_key: "1b7599748098fde9410a4cd4d8d7a792"
server_key_iv: "b2f3a17f96be00c8c47a98b1e8e88d44"
server_key_version: "encryptionkey@0"
I20260621 05:10:55.563726 11993 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782018616648578-3598-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782018616648578-3598-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782018616648578-3598-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 05:10:55.578148 11993 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 05:10:55.580500 11993 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 05:10:55.580632 11993 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 05:10:55.580830 11993 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 05:10:55.581177 11993 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 05:10:55.581214 11993 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 05:10:55.581238 11993 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 05:10:55.581249 11993 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 05:10:55.587553 11993 rpc_server.cc:307] RPC server started. Bound to: 127.3.131.130:37507
I20260621 05:10:55.587587 12121 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.131.130:37507 every 8 connection(s)
I20260621 05:10:55.588013 11993 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782018616648578-3598-0/minicluster-data/ts-1/data/info.pb
Jun 21 05:10:55 dist-test-slave-vrq1 krb5kdc[11752](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782018655, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.131.130@KRBTEST.COM for kudu/127.3.131.190@KRBTEST.COM
I20260621 05:10:55.595466 3598 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWZOx2_/build/release/bin/kudu as pid 11993
I20260621 05:10:55.595634 3598 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782018616648578-3598-0/minicluster-data/ts-1/wal/instance
I20260621 05:10:55.595876 3598 external_mini_cluster.cc:1428] Setting key 315fb35eaab2d7c36b2066fef2fd8db8
I20260621 05:10:55.598587 11837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:10:55.588912 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.131.130:50451 (local address 127.3.131.190:37467)
0621 05:10:55.588980 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0621 05:10:55.588983 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0621 05:10:55.589229 (+ 246us) server_negotiation.cc:407] Connection header received
0621 05:10:55.589827 (+ 598us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:10:55.589828 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:10:55.589838 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:10:55.589891 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:10:55.590868 (+ 977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:10:55.591434 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:10:55.592506 (+ 1072us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:10:55.592699 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:10:55.592762 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:10:55.595381 (+ 2619us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:10:55.595384 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:10:55.595384 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:10:55.595395 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:10:55.597082 (+ 1687us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:10:55.597546 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:10:55.597548 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:10:55.597549 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:10:55.597619 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:10:55.597892 (+ 273us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:10:55.597893 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:10:55.597893 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:10:55.598020 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:10:55.598103 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0621 05:10:55.598397 (+ 294us) server_negotiation.cc:299] Negotiation successful
0621 05:10:55.598486 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260621 05:10:55.598939 12124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:10:55.588910 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.131.190:37467 (local address 127.3.131.130:50451)
0621 05:10:55.589087 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0621 05:10:55.589138 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0621 05:10:55.589734 (+ 596us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:10:55.589953 (+ 219us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:10:55.589957 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:10:55.590559 (+ 602us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:10:55.590776 (+ 217us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:10:55.590781 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:10:55.591551 (+ 770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:10:55.591552 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:10:55.592396 (+ 844us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:10:55.592398 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:10:55.592813 (+ 415us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:10:55.592814 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:10:55.592934 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:10:55.593624 (+ 690us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 05:10:55.593633 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:10:55.595266 (+ 1633us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 05:10:55.597179 (+ 1913us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:10:55.597189 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:10:55.597191 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:10:55.597463 (+ 272us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:10:55.597692 (+ 229us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:10:55.597693 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:10:55.597693 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:10:55.597820 (+ 127us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:10:55.598094 (+ 274us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 05:10:55.598102 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 05:10:55.598261 (+ 159us) client_negotiation.cc:769] Sending connection context
0621 05:10:55.598374 (+ 113us) client_negotiation.cc:240] Negotiation successful
0621 05:10:55.598563 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":89,"threads_started":1}
I20260621 05:10:55.599500 12122 heartbeater.cc:347] Connected to a master server at 127.3.131.190:37467
I20260621 05:10:55.599624 12122 heartbeater.cc:464] Registering TS with master...
I20260621 05:10:55.599815 12122 heartbeater.cc:511] Master 127.3.131.190:37467 requested a full tablet report, sending...
I20260621 05:10:55.600312 11796 ts_manager.cc:194] Registered new tserver with Master: 83e89bf9905742ec80633a26819621ae (127.3.131.130:37507)
I20260621 05:10:55.600870 11796 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.131.130@KRBTEST.COM'} at 127.3.131.130:50451
WARNING: no policy specified for kudu/127.3.131.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.131.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.131.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782018616648578-3598-0/krb5kdc/kudu_127.3.131.131.keytab.
Entry for principal kudu/127.3.131.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782018616648578-3598-0/krb5kdc/kudu_127.3.131.131.keytab.
I20260621 05:10:55.623392 3598 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWZOx2_/build/release/bin/kudu
/tmp/dist-test-taskWZOx2_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782018616648578-3598-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782018616648578-3598-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782018616648578-3598-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782018616648578-3598-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.131.131:0
--local_ip_for_outbound_sockets=127.3.131.131
--webserver_interface=127.3.131.131
--webserver_port=0
--tserver_master_addrs=127.3.131.190:37467
--keytab_file=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782018616648578-3598-0/krb5kdc/kudu_127.3.131.131.keytab
--principal=kudu/127.3.131.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.131.148:42409
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782018616648578-3598-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782018616648578-3598-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782018616648578-3598-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 05:10:55.709261 12127 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 05:10:55.709477 12127 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 05:10:55.709506 12127 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 05:10:55.710719 12127 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 05:10:55.710760 12127 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 05:10:55.710788 12127 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.131.131
I20260621 05:10:55.712380 12127 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.131.148:42409
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782018616648578-3598-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782018616648578-3598-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782018616648578-3598-0/krb5kdc/kudu_127.3.131.131.keytab
--principal=kudu/127.3.131.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.131.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782018616648578-3598-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.131.131
--webserver_port=0
--tserver_master_addrs=127.3.131.190:37467
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12127
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.131.131
--log_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782018616648578-3598-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 21 Jun 2026 04:16:16 UTC on ccycloud.quasar-xycaha.root.comops.site
build id 4457881
I20260621 05:10:55.712646 12127 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 05:10:55.712886 12127 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 05:10:55.718143 12132 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:10:55.718217 12133 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:10:55.718216 12135 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:10:55.718269 12127 server_base.cc:1034] running on GCE node
I20260621 05:10:55.718577 12127 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 05:10:55.718809 12127 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 05:10:55.719974 12127 hybrid_clock.cc:648] HybridClock initialized: now 1782018655719953 us; error 40 us; skew 500 ppm
Jun 21 05:10:55 dist-test-slave-vrq1 krb5kdc[11752](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782018655, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.131.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 05:10:55.722591 12127 init.cc:377] Logged in from keytab as kudu/127.3.131.131@KRBTEST.COM (short username kudu)
I20260621 05:10:55.723065 12127 webserver.cc:460] Webserver started at http://127.3.131.131:35899/ using document root <none> and password file <none>
I20260621 05:10:55.723340 12127 fs_manager.cc:359] Metadata directory not provided
I20260621 05:10:55.723390 12127 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 05:10:55.723498 12127 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 05:10:55.724572 12127 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782018616648578-3598-0/minicluster-data/ts-2/data/instance:
uuid: "291a7808d0124955b40dd02c134d409b"
format_stamp: "Formatted at 2026-06-21 05:10:55 on dist-test-slave-vrq1"
server_key: "3a9c4fd204b31aa54d0b56b44110bcbc"
server_key_iv: "870a28fd993e4899e48880c27ecbefe2"
server_key_version: "encryptionkey@0"
I20260621 05:10:55.724928 12127 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782018616648578-3598-0/minicluster-data/ts-2/wal/instance:
uuid: "291a7808d0124955b40dd02c134d409b"
format_stamp: "Formatted at 2026-06-21 05:10:55 on dist-test-slave-vrq1"
server_key: "3a9c4fd204b31aa54d0b56b44110bcbc"
server_key_iv: "870a28fd993e4899e48880c27ecbefe2"
server_key_version: "encryptionkey@0"
I20260621 05:10:55.726425 12127 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260621 05:10:55.727306 12142 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 05:10:55.727466 12127 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260621 05:10:55.727557 12127 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782018616648578-3598-0/minicluster-data/ts-2/data,/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782018616648578-3598-0/minicluster-data/ts-2/wal
uuid: "291a7808d0124955b40dd02c134d409b"
format_stamp: "Formatted at 2026-06-21 05:10:55 on dist-test-slave-vrq1"
server_key: "3a9c4fd204b31aa54d0b56b44110bcbc"
server_key_iv: "870a28fd993e4899e48880c27ecbefe2"
server_key_version: "encryptionkey@0"
I20260621 05:10:55.727618 12127 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782018616648578-3598-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782018616648578-3598-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782018616648578-3598-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 05:10:55.738278 12127 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 05:10:55.740334 12127 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 05:10:55.740446 12127 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 05:10:55.740633 12127 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 05:10:55.740906 12127 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 05:10:55.740937 12127 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 05:10:55.740960 12127 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 05:10:55.740973 12127 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 05:10:55.746484 12127 rpc_server.cc:307] RPC server started. Bound to: 127.3.131.131:41055
I20260621 05:10:55.746562 12255 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.131.131:41055 every 8 connection(s)
I20260621 05:10:55.746874 12127 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782018616648578-3598-0/minicluster-data/ts-2/data/info.pb
I20260621 05:10:55.748932 3598 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWZOx2_/build/release/bin/kudu as pid 12127
I20260621 05:10:55.749011 3598 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782018616648578-3598-0/minicluster-data/ts-2/wal/instance
I20260621 05:10:55.749223 3598 external_mini_cluster.cc:1428] Setting key 10b665f82e99308f67217c9e6b3a9696
Jun 21 05:10:55 dist-test-slave-vrq1 krb5kdc[11752](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782018655, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.131.131@KRBTEST.COM for kudu/127.3.131.190@KRBTEST.COM
I20260621 05:10:55.757241 11837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:10:55.747634 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.131.131:46489 (local address 127.3.131.190:37467)
0621 05:10:55.747692 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0621 05:10:55.747694 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:10:55.747885 (+ 191us) server_negotiation.cc:407] Connection header received
0621 05:10:55.748391 (+ 506us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:10:55.748392 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:10:55.748403 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:10:55.748467 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:10:55.749688 (+ 1221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:10:55.750349 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:10:55.751437 (+ 1088us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:10:55.751657 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:10:55.751751 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:10:55.754204 (+ 2453us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:10:55.754205 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:10:55.754206 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:10:55.754214 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:10:55.755575 (+ 1361us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:10:55.756063 (+ 488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:10:55.756065 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:10:55.756065 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:10:55.756155 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:10:55.756428 (+ 273us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:10:55.756429 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:10:55.756430 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:10:55.756554 (+ 124us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:10:55.756691 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0621 05:10:55.757045 (+ 354us) server_negotiation.cc:299] Negotiation successful
0621 05:10:55.757136 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260621 05:10:55.757535 12258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:10:55.747584 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.131.190:37467 (local address 127.3.131.131:46489)
0621 05:10:55.747752 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0621 05:10:55.747794 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0621 05:10:55.748287 (+ 493us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:10:55.748579 (+ 292us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:10:55.748583 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:10:55.749244 (+ 661us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:10:55.749588 (+ 344us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:10:55.749594 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:10:55.750436 (+ 842us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:10:55.750437 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:10:55.751349 (+ 912us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:10:55.751352 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:10:55.751761 (+ 409us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:10:55.751762 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:10:55.751908 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:10:55.752394 (+ 486us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 05:10:55.752402 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:10:55.754071 (+ 1669us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 05:10:55.755705 (+ 1634us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:10:55.755712 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:10:55.755714 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:10:55.755964 (+ 250us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:10:55.756232 (+ 268us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:10:55.756233 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:10:55.756234 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:10:55.756357 (+ 123us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:10:55.756683 (+ 326us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 05:10:55.756689 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 05:10:55.756882 (+ 193us) client_negotiation.cc:769] Sending connection context
0621 05:10:55.757022 (+ 140us) client_negotiation.cc:240] Negotiation successful
0621 05:10:55.757253 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":100,"threads_started":1}
I20260621 05:10:55.758059 12256 heartbeater.cc:347] Connected to a master server at 127.3.131.190:37467
I20260621 05:10:55.758157 12256 heartbeater.cc:464] Registering TS with master...
I20260621 05:10:55.758307 12256 heartbeater.cc:511] Master 127.3.131.190:37467 requested a full tablet report, sending...
I20260621 05:10:55.758896 11796 ts_manager.cc:194] Registered new tserver with Master: 291a7808d0124955b40dd02c134d409b (127.3.131.131:41055)
I20260621 05:10:55.759390 11796 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.131.131@KRBTEST.COM'} at 127.3.131.131:46489
I20260621 05:10:55.761210 3598 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.3.131.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.3.131.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.3.131.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782018616648578-3598-0/krb5kdc/oryx_127.3.131.129.keytab.
Entry for principal oryx/127.3.131.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782018616648578-3598-0/krb5kdc/oryx_127.3.131.129.keytab.
WARNING: no policy specified for oryx/127.3.131.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.3.131.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.3.131.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782018616648578-3598-0/krb5kdc/oryx_127.3.131.130.keytab.
Entry for principal oryx/127.3.131.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782018616648578-3598-0/krb5kdc/oryx_127.3.131.130.keytab.
WARNING: no policy specified for oryx/127.3.131.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.3.131.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.3.131.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782018616648578-3598-0/krb5kdc/oryx_127.3.131.131.keytab.
Entry for principal oryx/127.3.131.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782018616648578-3598-0/krb5kdc/oryx_127.3.131.131.keytab.
I20260621 05:10:55.842625 3598 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWZOx2_/build/release/bin/kudu with pid 11859
I20260621 05:10:55.847496 3598 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWZOx2_/build/release/bin/kudu with pid 11993
I20260621 05:10:55.852015 3598 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWZOx2_/build/release/bin/kudu with pid 12127
I20260621 05:10:55.856369 3598 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWZOx2_/build/release/bin/kudu with pid 11765
I20260621 05:10:55.862125 3598 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWZOx2_/build/release/bin/kudu
/tmp/dist-test-taskWZOx2_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782018616648578-3598-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782018616648578-3598-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782018616648578-3598-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782018616648578-3598-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.131.190:37467
--webserver_interface=127.3.131.190
--webserver_port=36615
--keytab_file=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782018616648578-3598-0/krb5kdc/kudu_127.3.131.190.keytab
--principal=kudu/127.3.131.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.131.148:42409
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.131.190:37467
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782018616648578-3598-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782018616648578-3598-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782018616648578-3598-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 05:10:55.944722 12265 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 05:10:55.944952 12265 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 05:10:55.944979 12265 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 05:10:55.946199 12265 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 05:10:55.946247 12265 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 05:10:55.946261 12265 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 05:10:55.946274 12265 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 05:10:55.946287 12265 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 05:10:55.947847 12265 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.131.148:42409
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782018616648578-3598-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782018616648578-3598-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.131.190:37467
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782018616648578-3598-0/krb5kdc/kudu_127.3.131.190.keytab
--principal=kudu/127.3.131.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.131.190:37467
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782018616648578-3598-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.131.190
--webserver_port=36615
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12265
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782018616648578-3598-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 21 Jun 2026 04:16:16 UTC on ccycloud.quasar-xycaha.root.comops.site
build id 4457881
I20260621 05:10:55.948115 12265 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 05:10:55.948367 12265 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 05:10:55.953682 12273 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:10:55.953715 12271 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:10:55.953682 12270 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:10:55.953986 12265 server_base.cc:1034] running on GCE node
I20260621 05:10:55.954249 12265 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 05:10:55.954541 12265 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 05:10:55.955758 12265 hybrid_clock.cc:648] HybridClock initialized: now 1782018655955744 us; error 41 us; skew 500 ppm
Jun 21 05:10:55 dist-test-slave-vrq1 krb5kdc[11752](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782018655, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.131.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 05:10:55.958683 12265 init.cc:377] Logged in from keytab as kudu/127.3.131.190@KRBTEST.COM (short username kudu)
I20260621 05:10:55.959276 12265 webserver.cc:460] Webserver started at http://127.3.131.190:36615/ using document root <none> and password file <none>
I20260621 05:10:55.959515 12265 fs_manager.cc:359] Metadata directory not provided
I20260621 05:10:55.959602 12265 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 05:10:55.961100 12265 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.002s
I20260621 05:10:55.961896 12280 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 05:10:55.962152 12265 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260621 05:10:55.962280 12265 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782018616648578-3598-0/minicluster-data/master-0/data,/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782018616648578-3598-0/minicluster-data/master-0/wal
uuid: "5ab2baa2ec874efdbbae3f8fc138ff73"
format_stamp: "Formatted at 2026-06-21 05:10:55 on dist-test-slave-vrq1"
server_key: "8893e8723530060035981ac73d9c20e4"
server_key_iv: "a4932200db912dc1037f9e7f0531a8ea"
server_key_version: "encryptionkey@0"
I20260621 05:10:55.962721 12265 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782018616648578-3598-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782018616648578-3598-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782018616648578-3598-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 05:10:55.979516 12265 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 05:10:55.982079 12265 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 05:10:55.982257 12265 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 05:10:55.986368 12265 rpc_server.cc:307] RPC server started. Bound to: 127.3.131.190:37467
I20260621 05:10:55.986440 12332 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.131.190:37467 every 8 connection(s)
I20260621 05:10:55.986888 12265 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782018616648578-3598-0/minicluster-data/master-0/data/info.pb
I20260621 05:10:55.987835 3598 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWZOx2_/build/release/bin/kudu as pid 12265
I20260621 05:10:55.988240 3598 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWZOx2_/build/release/bin/kudu
/tmp/dist-test-taskWZOx2_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782018616648578-3598-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782018616648578-3598-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782018616648578-3598-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782018616648578-3598-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.131.129:35715
--local_ip_for_outbound_sockets=127.3.131.129
--tserver_master_addrs=127.3.131.190:37467
--webserver_port=35113
--webserver_interface=127.3.131.129
--keytab_file=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782018616648578-3598-0/krb5kdc/kudu_127.3.131.129.keytab
--principal=kudu/127.3.131.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.131.148:42409
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.3.131.129
--keytab_file=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782018616648578-3598-0/krb5kdc/oryx_127.3.131.129.keytab with env {KRB5CCNAME=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782018616648578-3598-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782018616648578-3598-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782018616648578-3598-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260621 05:10:55.988937 12333 sys_catalog.cc:263] Verifying existing consensus state
I20260621 05:10:55.989735 12333 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5ab2baa2ec874efdbbae3f8fc138ff73: Bootstrap starting.
I20260621 05:10:55.998607 12333 log.cc:824] T 00000000000000000000000000000000 P 5ab2baa2ec874efdbbae3f8fc138ff73: Log is configured to *not* fsync() on all Append() calls
I20260621 05:10:56.000559 12333 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5ab2baa2ec874efdbbae3f8fc138ff73: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260621 05:10:56.000905 12333 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5ab2baa2ec874efdbbae3f8fc138ff73: Bootstrap complete.
I20260621 05:10:56.003616 12333 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5ab2baa2ec874efdbbae3f8fc138ff73 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ab2baa2ec874efdbbae3f8fc138ff73" member_type: VOTER last_known_addr { host: "127.3.131.190" port: 37467 } }
I20260621 05:10:56.003907 12333 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5ab2baa2ec874efdbbae3f8fc138ff73 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5ab2baa2ec874efdbbae3f8fc138ff73, State: Initialized, Role: FOLLOWER
I20260621 05:10:56.004149 12333 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5ab2baa2ec874efdbbae3f8fc138ff73 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ab2baa2ec874efdbbae3f8fc138ff73" member_type: VOTER last_known_addr { host: "127.3.131.190" port: 37467 } }
I20260621 05:10:56.004258 12333 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5ab2baa2ec874efdbbae3f8fc138ff73 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260621 05:10:56.004316 12333 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5ab2baa2ec874efdbbae3f8fc138ff73 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260621 05:10:56.004372 12333 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5ab2baa2ec874efdbbae3f8fc138ff73 [term 1 FOLLOWER]: Advancing to term 2
I20260621 05:10:56.005512 12333 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5ab2baa2ec874efdbbae3f8fc138ff73 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ab2baa2ec874efdbbae3f8fc138ff73" member_type: VOTER last_known_addr { host: "127.3.131.190" port: 37467 } }
I20260621 05:10:56.005652 12333 leader_election.cc:302] T 00000000000000000000000000000000 P 5ab2baa2ec874efdbbae3f8fc138ff73 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5ab2baa2ec874efdbbae3f8fc138ff73; no voters:
I20260621 05:10:56.005889 12333 leader_election.cc:288] T 00000000000000000000000000000000 P 5ab2baa2ec874efdbbae3f8fc138ff73 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260621 05:10:56.005932 12337 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5ab2baa2ec874efdbbae3f8fc138ff73 [term 2 FOLLOWER]: Leader election won for term 2
I20260621 05:10:56.006062 12337 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5ab2baa2ec874efdbbae3f8fc138ff73 [term 2 LEADER]: Becoming Leader. State: Replica: 5ab2baa2ec874efdbbae3f8fc138ff73, State: Running, Role: LEADER
I20260621 05:10:56.006233 12333 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5ab2baa2ec874efdbbae3f8fc138ff73 [sys.catalog]: configured and running, proceeding with master startup.
I20260621 05:10:56.006229 12337 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5ab2baa2ec874efdbbae3f8fc138ff73 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ab2baa2ec874efdbbae3f8fc138ff73" member_type: VOTER last_known_addr { host: "127.3.131.190" port: 37467 } }
I20260621 05:10:56.007819 12338 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5ab2baa2ec874efdbbae3f8fc138ff73 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "5ab2baa2ec874efdbbae3f8fc138ff73" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ab2baa2ec874efdbbae3f8fc138ff73" member_type: VOTER last_known_addr { host: "127.3.131.190" port: 37467 } } }
I20260621 05:10:56.008122 12338 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5ab2baa2ec874efdbbae3f8fc138ff73 [sys.catalog]: This master's current role is: LEADER
I20260621 05:10:56.008262 12351 catalog_manager.cc:1259] Loaded cluster ID: 6dd7dd2b0d704f138f788517f9796ba6
I20260621 05:10:56.007956 12339 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5ab2baa2ec874efdbbae3f8fc138ff73 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5ab2baa2ec874efdbbae3f8fc138ff73. Latest consensus state: current_term: 2 leader_uuid: "5ab2baa2ec874efdbbae3f8fc138ff73" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ab2baa2ec874efdbbae3f8fc138ff73" member_type: VOTER last_known_addr { host: "127.3.131.190" port: 37467 } } }
I20260621 05:10:56.008343 12351 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 5ab2baa2ec874efdbbae3f8fc138ff73: loading cluster ID for follower catalog manager: success
I20260621 05:10:56.008358 12339 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5ab2baa2ec874efdbbae3f8fc138ff73 [sys.catalog]: This master's current role is: LEADER
I20260621 05:10:56.009826 12351 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 5ab2baa2ec874efdbbae3f8fc138ff73: acquiring CA information for follower catalog manager: success
I20260621 05:10:56.010210 12351 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 5ab2baa2ec874efdbbae3f8fc138ff73: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260621 05:10:56.010298 12353 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260621 05:10:56.010538 12353 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260621 05:10:56.010623 12353 catalog_manager.cc:1259] Loaded cluster ID: 6dd7dd2b0d704f138f788517f9796ba6
I20260621 05:10:56.010658 12353 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260621 05:10:56.010833 12353 catalog_manager.cc:1504] Loading token signing keys...
I20260621 05:10:56.010936 12353 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 5ab2baa2ec874efdbbae3f8fc138ff73: Loaded TSK: 0
I20260621 05:10:56.011066 12353 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260621 05:10:56.078064 12335 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 05:10:56.078342 12335 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 05:10:56.078388 12335 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 05:10:56.079905 12335 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 05:10:56.079984 12335 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 05:10:56.080039 12335 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.131.129
I20260621 05:10:56.081789 12335 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.131.148:42409
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782018616648578-3598-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782018616648578-3598-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782018616648578-3598-0/krb5kdc/oryx_127.3.131.129.keytab
--principal=oryx/127.3.131.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.131.129:35715
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782018616648578-3598-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.131.129
--webserver_port=35113
--tserver_master_addrs=127.3.131.190:37467
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12335
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.131.129
--log_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782018616648578-3598-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 21 Jun 2026 04:16:16 UTC on ccycloud.quasar-xycaha.root.comops.site
build id 4457881
I20260621 05:10:56.082119 12335 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 05:10:56.082387 12335 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 05:10:56.088811 12360 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:10:56.088817 12362 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:10:56.088879 12335 server_base.cc:1034] running on GCE node
W20260621 05:10:56.088816 12359 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:10:56.089341 12335 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 05:10:56.089620 12335 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 05:10:56.090808 12335 hybrid_clock.cc:648] HybridClock initialized: now 1782018656090792 us; error 31 us; skew 500 ppm
Jun 21 05:10:56 dist-test-slave-vrq1 krb5kdc[11752](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782018656, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.131.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 05:10:56.093760 12335 init.cc:377] Logged in from keytab as oryx/127.3.131.129@KRBTEST.COM (short username oryx)
I20260621 05:10:56.094316 12335 webserver.cc:460] Webserver started at http://127.3.131.129:35113/ using document root <none> and password file <none>
I20260621 05:10:56.094596 12335 fs_manager.cc:359] Metadata directory not provided
I20260621 05:10:56.094681 12335 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 05:10:56.096653 12335 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260621 05:10:56.097409 12369 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 05:10:56.097589 12335 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260621 05:10:56.097682 12335 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782018616648578-3598-0/minicluster-data/ts-0/data,/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782018616648578-3598-0/minicluster-data/ts-0/wal
uuid: "4623cc8171b7486bb1b0d5bdcd71fdd5"
format_stamp: "Formatted at 2026-06-21 05:10:55 on dist-test-slave-vrq1"
server_key: "8f18b0ad387973f1fc97692097d7c285"
server_key_iv: "5db7fb9a2f33e7d1603f8fa357d41ebb"
server_key_version: "encryptionkey@0"
I20260621 05:10:56.098029 12335 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782018616648578-3598-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782018616648578-3598-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782018616648578-3598-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 05:10:56.108255 12335 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 05:10:56.110671 12335 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 05:10:56.110823 12335 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 05:10:56.111120 12335 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 05:10:56.111470 12335 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 05:10:56.111527 12335 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 05:10:56.111579 12335 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 05:10:56.111613 12335 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 05:10:56.118036 12482 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.131.129:35715 every 8 connection(s)
I20260621 05:10:56.118124 12335 rpc_server.cc:307] RPC server started. Bound to: 127.3.131.129:35715
I20260621 05:10:56.118517 12335 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782018616648578-3598-0/minicluster-data/ts-0/data/info.pb
I20260621 05:10:56.123972 3598 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWZOx2_/build/release/bin/kudu as pid 12335
I20260621 05:10:56.124316 3598 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWZOx2_/build/release/bin/kudu
/tmp/dist-test-taskWZOx2_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782018616648578-3598-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782018616648578-3598-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782018616648578-3598-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782018616648578-3598-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.131.130:37507
--local_ip_for_outbound_sockets=127.3.131.130
--tserver_master_addrs=127.3.131.190:37467
--webserver_port=40481
--webserver_interface=127.3.131.130
--keytab_file=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782018616648578-3598-0/krb5kdc/kudu_127.3.131.130.keytab
--principal=kudu/127.3.131.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.131.148:42409
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.3.131.130
--keytab_file=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782018616648578-3598-0/krb5kdc/oryx_127.3.131.130.keytab with env {KRB5CCNAME=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782018616648578-3598-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782018616648578-3598-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782018616648578-3598-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 21 05:10:56 dist-test-slave-vrq1 krb5kdc[11752](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.131.129@KRBTEST.COM for oryx/127.3.131.190@KRBTEST.COM, Server not found in Kerberos database
Jun 21 05:10:56 dist-test-slave-vrq1 krb5kdc[11752](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.131.129@KRBTEST.COM for oryx/127.3.131.190@KRBTEST.COM, Server not found in Kerberos database
I20260621 05:10:56.129256 12485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:10:56.119408 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.131.190:37467 (local address 127.3.131.129:45037)
0621 05:10:56.119603 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0621 05:10:56.119648 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0621 05:10:56.120123 (+ 475us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:10:56.120645 (+ 522us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:10:56.120648 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:10:56.121266 (+ 618us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:10:56.121502 (+ 236us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:10:56.121507 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:10:56.123316 (+ 1809us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:10:56.123318 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:10:56.124461 (+ 1143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:10:56.124467 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:10:56.125163 (+ 696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:10:56.125164 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:10:56.125298 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:10:56.125937 (+ 639us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 05:10:56.125950 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:10:56.129013 (+ 3063us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.131.190:37467: Server oryx/127.3.131.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":101,"threads_started":1}
W20260621 05:10:56.129319 12486 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 05:10:56.119505 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.131.129:45037 (local address 127.3.131.190:37467)
0621 05:10:56.119737 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0621 05:10:56.119746 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0621 05:10:56.119784 (+ 38us) server_negotiation.cc:407] Connection header received
0621 05:10:56.120335 (+ 551us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:10:56.120343 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:10:56.120405 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:10:56.120587 (+ 182us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:10:56.121648 (+ 1061us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:10:56.123125 (+ 1477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:10:56.124586 (+ 1461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:10:56.125056 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:10:56.125186 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:10:56.129157 (+ 3971us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.131.129:45037: BlockingRecv error: recv got EOF from 127.3.131.129:45037 (error 108)
Metrics: {"server-negotiator.queue_time_us":186,"thread_start_us":118,"threads_started":1}
W20260621 05:10:56.129647 12483 heartbeater.cc:644] Failed to heartbeat to 127.3.131.190:37467 (0 consecutive failures): Not authorized: Failed to ping master at 127.3.131.190:37467: Client connection negotiation failed: client connection to 127.3.131.190:37467: Server oryx/127.3.131.190@KRBTEST.COM not found in Kerberos database
Jun 21 05:10:56 dist-test-slave-vrq1 krb5kdc[11752](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.131.129@KRBTEST.COM for oryx/127.3.131.190@KRBTEST.COM, Server not found in Kerberos database
Jun 21 05:10:56 dist-test-slave-vrq1 krb5kdc[11752](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.131.129@KRBTEST.COM for oryx/127.3.131.190@KRBTEST.COM, Server not found in Kerberos database
I20260621 05:10:56.134560 12485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:10:56.129960 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.131.190:37467 (local address 127.3.131.129:56969)
0621 05:10:56.130022 (+ 62us) negotiation.cc:107] Waiting for socket to connect
0621 05:10:56.130033 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0621 05:10:56.130079 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:10:56.130256 (+ 177us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:10:56.130257 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:10:56.130408 (+ 151us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:10:56.130466 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:10:56.130468 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:10:56.131140 (+ 672us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:10:56.131141 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:10:56.131735 (+ 594us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:10:56.131737 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:10:56.132114 (+ 377us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:10:56.132115 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:10:56.132187 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:10:56.132659 (+ 472us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 05:10:56.132664 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:10:56.134462 (+ 1798us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.131.190:37467: Server oryx/127.3.131.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":34}
W20260621 05:10:56.134629 12486 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 05:10:56.130013 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.131.129:56969 (local address 127.3.131.190:37467)
0621 05:10:56.130051 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0621 05:10:56.130053 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:10:56.130063 (+ 10us) server_negotiation.cc:407] Connection header received
0621 05:10:56.130164 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:10:56.130165 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:10:56.130176 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:10:56.130250 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:10:56.130561 (+ 311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:10:56.131059 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:10:56.131824 (+ 765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:10:56.132028 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:10:56.132113 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:10:56.134559 (+ 2446us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.131.129:56969: BlockingRecv error: recv got EOF from 127.3.131.129:56969 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
Jun 21 05:10:56 dist-test-slave-vrq1 krb5kdc[11752](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.131.129@KRBTEST.COM for oryx/127.3.131.190@KRBTEST.COM, Server not found in Kerberos database
Jun 21 05:10:56 dist-test-slave-vrq1 krb5kdc[11752](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.131.129@KRBTEST.COM for oryx/127.3.131.190@KRBTEST.COM, Server not found in Kerberos database
I20260621 05:10:56.139598 12485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:10:56.135180 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.131.190:37467 (local address 127.3.131.129:48981)
0621 05:10:56.135286 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0621 05:10:56.135298 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0621 05:10:56.135348 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:10:56.135500 (+ 152us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:10:56.135501 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:10:56.135662 (+ 161us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:10:56.135706 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:10:56.135708 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:10:56.136369 (+ 661us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:10:56.136370 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:10:56.136868 (+ 498us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:10:56.136870 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:10:56.137225 (+ 355us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:10:56.137226 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:10:56.137287 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:10:56.137730 (+ 443us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 05:10:56.137735 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:10:56.139507 (+ 1772us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.131.190:37467: Server oryx/127.3.131.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":90}
W20260621 05:10:56.139652 12486 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 05:10:56.135248 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.131.129:48981 (local address 127.3.131.190:37467)
0621 05:10:56.135293 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0621 05:10:56.135295 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:10:56.135331 (+ 36us) server_negotiation.cc:407] Connection header received
0621 05:10:56.135422 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:10:56.135423 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:10:56.135432 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:10:56.135496 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:10:56.135798 (+ 302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:10:56.136282 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:10:56.136950 (+ 668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:10:56.137143 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:10:56.137216 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:10:56.139586 (+ 2370us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.131.129:48981: BlockingRecv error: recv got EOF from 127.3.131.129:48981 (error 108)
Metrics: {"server-negotiator.queue_time_us":17}
W20260621 05:10:56.140061 12483 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260621 05:10:56.219790 12487 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 05:10:56.220047 12487 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 05:10:56.220077 12487 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 05:10:56.221335 12487 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 05:10:56.221378 12487 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 05:10:56.221407 12487 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.131.130
I20260621 05:10:56.222930 12487 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.131.148:42409
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782018616648578-3598-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782018616648578-3598-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782018616648578-3598-0/krb5kdc/oryx_127.3.131.130.keytab
--principal=oryx/127.3.131.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.131.130:37507
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782018616648578-3598-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.131.130
--webserver_port=40481
--tserver_master_addrs=127.3.131.190:37467
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12487
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.131.130
--log_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782018616648578-3598-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 21 Jun 2026 04:16:16 UTC on ccycloud.quasar-xycaha.root.comops.site
build id 4457881
I20260621 05:10:56.223152 12487 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 05:10:56.223435 12487 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 05:10:56.228760 12492 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:10:56.228760 12493 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:10:56.228801 12495 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:10:56.229344 12487 server_base.cc:1034] running on GCE node
I20260621 05:10:56.229662 12487 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 05:10:56.229940 12487 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 05:10:56.231156 12487 hybrid_clock.cc:648] HybridClock initialized: now 1782018656231141 us; error 29 us; skew 500 ppm
Jun 21 05:10:56 dist-test-slave-vrq1 krb5kdc[11752](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782018656, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.131.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 05:10:56.234205 12487 init.cc:377] Logged in from keytab as oryx/127.3.131.130@KRBTEST.COM (short username oryx)
I20260621 05:10:56.234880 12487 webserver.cc:460] Webserver started at http://127.3.131.130:40481/ using document root <none> and password file <none>
I20260621 05:10:56.235301 12487 fs_manager.cc:359] Metadata directory not provided
I20260621 05:10:56.235400 12487 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 05:10:56.236804 12487 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260621 05:10:56.237474 12502 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 05:10:56.237747 12487 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260621 05:10:56.237835 12487 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782018616648578-3598-0/minicluster-data/ts-1/data,/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782018616648578-3598-0/minicluster-data/ts-1/wal
uuid: "83e89bf9905742ec80633a26819621ae"
format_stamp: "Formatted at 2026-06-21 05:10:55 on dist-test-slave-vrq1"
server_key: "1b7599748098fde9410a4cd4d8d7a792"
server_key_iv: "b2f3a17f96be00c8c47a98b1e8e88d44"
server_key_version: "encryptionkey@0"
I20260621 05:10:56.238227 12487 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782018616648578-3598-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782018616648578-3598-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782018616648578-3598-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 05:10:56.254793 12487 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 05:10:56.257232 12487 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 05:10:56.257424 12487 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 05:10:56.257688 12487 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 05:10:56.258004 12487 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 05:10:56.258059 12487 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 05:10:56.258100 12487 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 05:10:56.258128 12487 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 05:10:56.264633 12487 rpc_server.cc:307] RPC server started. Bound to: 127.3.131.130:37507
I20260621 05:10:56.264691 12615 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.131.130:37507 every 8 connection(s)
I20260621 05:10:56.265077 12487 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782018616648578-3598-0/minicluster-data/ts-1/data/info.pb
I20260621 05:10:56.271667 3598 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWZOx2_/build/release/bin/kudu as pid 12487
I20260621 05:10:56.271966 3598 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWZOx2_/build/release/bin/kudu
/tmp/dist-test-taskWZOx2_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782018616648578-3598-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782018616648578-3598-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782018616648578-3598-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782018616648578-3598-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.131.131:41055
--local_ip_for_outbound_sockets=127.3.131.131
--tserver_master_addrs=127.3.131.190:37467
--webserver_port=35899
--webserver_interface=127.3.131.131
--keytab_file=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782018616648578-3598-0/krb5kdc/kudu_127.3.131.131.keytab
--principal=kudu/127.3.131.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.131.148:42409
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.3.131.131
--keytab_file=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782018616648578-3598-0/krb5kdc/oryx_127.3.131.131.keytab with env {KRB5CCNAME=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782018616648578-3598-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782018616648578-3598-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782018616648578-3598-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 21 05:10:56 dist-test-slave-vrq1 krb5kdc[11752](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.131.130@KRBTEST.COM for oryx/127.3.131.190@KRBTEST.COM, Server not found in Kerberos database
Jun 21 05:10:56 dist-test-slave-vrq1 krb5kdc[11752](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.131.130@KRBTEST.COM for oryx/127.3.131.190@KRBTEST.COM, Server not found in Kerberos database
W20260621 05:10:56.273881 12486 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 05:10:56.266009 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.131.130:52891 (local address 127.3.131.190:37467)
0621 05:10:56.266088 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0621 05:10:56.266091 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0621 05:10:56.266339 (+ 248us) server_negotiation.cc:407] Connection header received
0621 05:10:56.266938 (+ 599us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:10:56.266939 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:10:56.266956 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:10:56.267011 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:10:56.268288 (+ 1277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:10:56.268816 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:10:56.269896 (+ 1080us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:10:56.270097 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:10:56.270170 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:10:56.273824 (+ 3654us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.131.130:52891: BlockingRecv error: recv got EOF from 127.3.131.130:52891 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
I20260621 05:10:56.273952 12618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:10:56.265986 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.131.190:37467 (local address 127.3.131.130:52891)
0621 05:10:56.266190 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0621 05:10:56.266235 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0621 05:10:56.266828 (+ 593us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:10:56.267128 (+ 300us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:10:56.267131 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:10:56.267780 (+ 649us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:10:56.267994 (+ 214us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:10:56.267999 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:10:56.268926 (+ 927us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:10:56.268927 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:10:56.269791 (+ 864us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:10:56.269794 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:10:56.270190 (+ 396us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:10:56.270191 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:10:56.270334 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:10:56.270861 (+ 527us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 05:10:56.270870 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:10:56.273742 (+ 2872us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.131.190:37467: Server oryx/127.3.131.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":161,"thread_start_us":73,"threads_started":1}
W20260621 05:10:56.274364 12616 heartbeater.cc:644] Failed to heartbeat to 127.3.131.190:37467 (0 consecutive failures): Not authorized: Failed to ping master at 127.3.131.190:37467: Client connection negotiation failed: client connection to 127.3.131.190:37467: Server oryx/127.3.131.190@KRBTEST.COM not found in Kerberos database
Jun 21 05:10:56 dist-test-slave-vrq1 krb5kdc[11752](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.131.130@KRBTEST.COM for oryx/127.3.131.190@KRBTEST.COM, Server not found in Kerberos database
Jun 21 05:10:56 dist-test-slave-vrq1 krb5kdc[11752](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.131.130@KRBTEST.COM for oryx/127.3.131.190@KRBTEST.COM, Server not found in Kerberos database
I20260621 05:10:56.279671 12618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:10:56.274676 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.131.190:37467 (local address 127.3.131.130:51015)
0621 05:10:56.274777 (+ 101us) negotiation.cc:107] Waiting for socket to connect
0621 05:10:56.274788 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0621 05:10:56.274841 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:10:56.275009 (+ 168us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:10:56.275010 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:10:56.275153 (+ 143us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:10:56.275245 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:10:56.275247 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:10:56.275971 (+ 724us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:10:56.275972 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:10:56.276482 (+ 510us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:10:56.276483 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:10:56.276848 (+ 365us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:10:56.276849 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:10:56.276913 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:10:56.277425 (+ 512us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 05:10:56.277430 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:10:56.279568 (+ 2138us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.131.190:37467: Server oryx/127.3.131.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":84}
W20260621 05:10:56.279754 12486 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 05:10:56.274733 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.131.130:51015 (local address 127.3.131.190:37467)
0621 05:10:56.274775 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0621 05:10:56.274777 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:10:56.274824 (+ 47us) server_negotiation.cc:407] Connection header received
0621 05:10:56.274906 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:10:56.274907 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:10:56.274922 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:10:56.275002 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:10:56.275343 (+ 341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:10:56.275884 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:10:56.276575 (+ 691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:10:56.276776 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:10:56.276852 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:10:56.279677 (+ 2825us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.131.130:51015: BlockingRecv error: recv got EOF from 127.3.131.130:51015 (error 108)
Metrics: {"server-negotiator.queue_time_us":17}
Jun 21 05:10:56 dist-test-slave-vrq1 krb5kdc[11752](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.131.130@KRBTEST.COM for oryx/127.3.131.190@KRBTEST.COM, Server not found in Kerberos database
Jun 21 05:10:56 dist-test-slave-vrq1 krb5kdc[11752](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.131.130@KRBTEST.COM for oryx/127.3.131.190@KRBTEST.COM, Server not found in Kerberos database
I20260621 05:10:56.285849 12618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:10:56.280236 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.131.190:37467 (local address 127.3.131.130:42779)
0621 05:10:56.280387 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0621 05:10:56.280399 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0621 05:10:56.280450 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:10:56.280619 (+ 169us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:10:56.280620 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:10:56.280783 (+ 163us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:10:56.280827 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:10:56.280829 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:10:56.281527 (+ 698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:10:56.281529 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:10:56.282442 (+ 913us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:10:56.282445 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:10:56.283065 (+ 620us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:10:56.283066 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:10:56.283134 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:10:56.283674 (+ 540us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 05:10:56.283679 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:10:56.285773 (+ 2094us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.131.190:37467: Server oryx/127.3.131.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":129}
W20260621 05:10:56.285961 12486 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 05:10:56.280322 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.131.130:42779 (local address 127.3.131.190:37467)
0621 05:10:56.280414 (+ 92us) server_negotiation.cc:206] Beginning negotiation
0621 05:10:56.280416 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:10:56.280434 (+ 18us) server_negotiation.cc:407] Connection header received
0621 05:10:56.280516 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:10:56.280517 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:10:56.280526 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:10:56.280590 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:10:56.280956 (+ 366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:10:56.281442 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:10:56.282553 (+ 1111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:10:56.282859 (+ 306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:10:56.282943 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:10:56.285877 (+ 2934us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.131.130:42779: BlockingRecv error: recv got EOF from 127.3.131.130:42779 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
W20260621 05:10:56.286091 12616 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260621 05:10:56.357993 12619 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 05:10:56.358219 12619 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 05:10:56.358245 12619 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 05:10:56.359525 12619 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 05:10:56.359572 12619 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 05:10:56.359601 12619 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.131.131
I20260621 05:10:56.361153 12619 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.131.148:42409
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782018616648578-3598-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782018616648578-3598-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782018616648578-3598-0/krb5kdc/oryx_127.3.131.131.keytab
--principal=oryx/127.3.131.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.131.131:41055
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782018616648578-3598-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.131.131
--webserver_port=35899
--tserver_master_addrs=127.3.131.190:37467
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12619
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.131.131
--log_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782018616648578-3598-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 21 Jun 2026 04:16:16 UTC on ccycloud.quasar-xycaha.root.comops.site
build id 4457881
I20260621 05:10:56.361373 12619 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 05:10:56.361558 12619 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 05:10:56.367197 12624 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:10:56.367256 12619 server_base.cc:1034] running on GCE node
W20260621 05:10:56.367272 12627 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:10:56.367229 12625 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:10:56.367730 12619 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 05:10:56.368005 12619 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 05:10:56.369163 12619 hybrid_clock.cc:648] HybridClock initialized: now 1782018656369138 us; error 37 us; skew 500 ppm
Jun 21 05:10:56 dist-test-slave-vrq1 krb5kdc[11752](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782018656, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.131.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 05:10:56.371908 12619 init.cc:377] Logged in from keytab as oryx/127.3.131.131@KRBTEST.COM (short username oryx)
I20260621 05:10:56.372359 12619 webserver.cc:460] Webserver started at http://127.3.131.131:35899/ using document root <none> and password file <none>
I20260621 05:10:56.372622 12619 fs_manager.cc:359] Metadata directory not provided
I20260621 05:10:56.372711 12619 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 05:10:56.374209 12619 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260621 05:10:56.374909 12634 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 05:10:56.375095 12619 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260621 05:10:56.375259 12619 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782018616648578-3598-0/minicluster-data/ts-2/data,/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782018616648578-3598-0/minicluster-data/ts-2/wal
uuid: "291a7808d0124955b40dd02c134d409b"
format_stamp: "Formatted at 2026-06-21 05:10:55 on dist-test-slave-vrq1"
server_key: "3a9c4fd204b31aa54d0b56b44110bcbc"
server_key_iv: "870a28fd993e4899e48880c27ecbefe2"
server_key_version: "encryptionkey@0"
I20260621 05:10:56.375641 12619 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782018616648578-3598-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782018616648578-3598-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782018616648578-3598-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 05:10:56.413301 12619 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 05:10:56.415483 12619 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 05:10:56.415668 12619 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 05:10:56.415951 12619 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 05:10:56.416275 12619 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 05:10:56.416330 12619 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 05:10:56.416374 12619 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 05:10:56.416402 12619 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 05:10:56.422385 12619 rpc_server.cc:307] RPC server started. Bound to: 127.3.131.131:41055
I20260621 05:10:56.422428 12747 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.131.131:41055 every 8 connection(s)
I20260621 05:10:56.422822 12619 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782018616648578-3598-0/minicluster-data/ts-2/data/info.pb
I20260621 05:10:56.427919 3598 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWZOx2_/build/release/bin/kudu as pid 12619
Jun 21 05:10:56 dist-test-slave-vrq1 krb5kdc[11752](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.131.131@KRBTEST.COM for oryx/127.3.131.190@KRBTEST.COM, Server not found in Kerberos database
Jun 21 05:10:56 dist-test-slave-vrq1 krb5kdc[11752](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.131.131@KRBTEST.COM for oryx/127.3.131.190@KRBTEST.COM, Server not found in Kerberos database
W20260621 05:10:56.430743 12486 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 05:10:56.423748 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.131.131:50179 (local address 127.3.131.190:37467)
0621 05:10:56.423797 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0621 05:10:56.423799 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:10:56.423896 (+ 97us) server_negotiation.cc:407] Connection header received
0621 05:10:56.424387 (+ 491us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:10:56.424388 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:10:56.424397 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:10:56.424459 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:10:56.425472 (+ 1013us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:10:56.425962 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:10:56.427032 (+ 1070us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:10:56.427324 (+ 292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:10:56.427380 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:10:56.430681 (+ 3301us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.131.131:50179: BlockingRecv error: recv got EOF from 127.3.131.131:50179 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
I20260621 05:10:56.430769 12750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:10:56.423607 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.131.190:37467 (local address 127.3.131.131:50179)
0621 05:10:56.423776 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0621 05:10:56.423819 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0621 05:10:56.424297 (+ 478us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:10:56.424537 (+ 240us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:10:56.424541 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:10:56.425175 (+ 634us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:10:56.425370 (+ 195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:10:56.425376 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:10:56.426061 (+ 685us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:10:56.426062 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:10:56.426948 (+ 886us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:10:56.426950 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:10:56.427404 (+ 454us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:10:56.427405 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:10:56.427523 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:10:56.428117 (+ 594us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 05:10:56.428126 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:10:56.430557 (+ 2431us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.131.190:37467: Server oryx/127.3.131.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":98,"threads_started":1}
W20260621 05:10:56.430970 12748 heartbeater.cc:644] Failed to heartbeat to 127.3.131.190:37467 (0 consecutive failures): Not authorized: Failed to ping master at 127.3.131.190:37467: Client connection negotiation failed: client connection to 127.3.131.190:37467: Server oryx/127.3.131.190@KRBTEST.COM not found in Kerberos database
Jun 21 05:10:56 dist-test-slave-vrq1 krb5kdc[11752](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.131.131@KRBTEST.COM for oryx/127.3.131.190@KRBTEST.COM, Server not found in Kerberos database
Jun 21 05:10:56 dist-test-slave-vrq1 krb5kdc[11752](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.131.131@KRBTEST.COM for oryx/127.3.131.190@KRBTEST.COM, Server not found in Kerberos database
I20260621 05:10:56.435840 12750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:10:56.431254 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.131.190:37467 (local address 127.3.131.131:50581)
0621 05:10:56.431311 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0621 05:10:56.431324 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0621 05:10:56.431371 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:10:56.431525 (+ 154us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:10:56.431525 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:10:56.431688 (+ 163us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:10:56.431734 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:10:56.431757 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:10:56.432391 (+ 634us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:10:56.432393 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:10:56.432981 (+ 588us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:10:56.432984 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:10:56.433345 (+ 361us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:10:56.433346 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:10:56.433411 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:10:56.433893 (+ 482us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 05:10:56.433897 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:10:56.435643 (+ 1746us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.131.190:37467: Server oryx/127.3.131.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":37}
W20260621 05:10:56.435885 12486 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 05:10:56.431332 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.131.131:50581 (local address 127.3.131.190:37467)
0621 05:10:56.431369 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0621 05:10:56.431370 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0621 05:10:56.431379 (+ 9us) server_negotiation.cc:407] Connection header received
0621 05:10:56.431434 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:10:56.431435 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:10:56.431442 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:10:56.431498 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:10:56.431845 (+ 347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:10:56.432312 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:10:56.433076 (+ 764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:10:56.433273 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:10:56.433346 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:10:56.435821 (+ 2475us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.131.131:50581: BlockingRecv error: recv got EOF from 127.3.131.131:50581 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
I20260621 05:10:56.436028 12752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:10:56.428484 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53248 (local address 127.3.131.190:37467)
0621 05:10:56.428634 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0621 05:10:56.428636 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:10:56.428653 (+ 17us) server_negotiation.cc:407] Connection header received
0621 05:10:56.428676 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:10:56.428677 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:10:56.428689 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:10:56.428766 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:10:56.429365 (+ 599us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:10:56.429852 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:10:56.430702 (+ 850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:10:56.431281 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:10:56.431351 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:10:56.432602 (+ 1251us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:10:56.432607 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:10:56.432607 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:10:56.432618 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:10:56.434428 (+ 1810us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:10:56.434875 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:10:56.434878 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:10:56.434879 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:10:56.434982 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:10:56.435216 (+ 234us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:10:56.435217 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:10:56.435217 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:10:56.435451 (+ 234us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:10:56.435611 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0621 05:10:56.435788 (+ 177us) server_negotiation.cc:299] Negotiation successful
0621 05:10:56.435911 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":128,"thread_start_us":74,"threads_started":1}
Jun 21 05:10:56 dist-test-slave-vrq1 krb5kdc[11752](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.131.131@KRBTEST.COM for oryx/127.3.131.190@KRBTEST.COM, Server not found in Kerberos database
Jun 21 05:10:56 dist-test-slave-vrq1 krb5kdc[11752](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.131.131@KRBTEST.COM for oryx/127.3.131.190@KRBTEST.COM, Server not found in Kerberos database
I20260621 05:10:56.441123 12750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:10:56.436575 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.131.190:37467 (local address 127.3.131.131:55475)
0621 05:10:56.436631 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0621 05:10:56.436642 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0621 05:10:56.436680 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:10:56.436964 (+ 284us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:10:56.436965 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:10:56.437094 (+ 129us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:10:56.437122 (+ 28us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:10:56.437123 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:10:56.437710 (+ 587us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:10:56.437710 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:10:56.438229 (+ 519us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:10:56.438230 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:10:56.438606 (+ 376us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:10:56.438607 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:10:56.438655 (+ 48us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:10:56.439161 (+ 506us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 05:10:56.439166 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:10:56.441069 (+ 1903us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.131.190:37467: Server oryx/127.3.131.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":38}
W20260621 05:10:56.441396 12752 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 05:10:56.436616 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.131.131:55475 (local address 127.3.131.190:37467)
0621 05:10:56.436656 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0621 05:10:56.436658 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:10:56.436669 (+ 11us) server_negotiation.cc:407] Connection header received
0621 05:10:56.436783 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:10:56.436788 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:10:56.436798 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:10:56.436860 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:10:56.437175 (+ 315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:10:56.437646 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:10:56.438307 (+ 661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:10:56.438492 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:10:56.438553 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:10:56.441288 (+ 2735us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.131.131:55475: BlockingRecv error: recv got EOF from 127.3.131.131:55475 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
W20260621 05:10:56.441434 12748 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Jun 21 05:10:57 dist-test-slave-vrq1 krb5kdc[11752](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.131.129@KRBTEST.COM for oryx/127.3.131.190@KRBTEST.COM, Server not found in Kerberos database
Jun 21 05:10:57 dist-test-slave-vrq1 krb5kdc[11752](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.131.129@KRBTEST.COM for oryx/127.3.131.190@KRBTEST.COM, Server not found in Kerberos database
I20260621 05:10:57.145668 12753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:10:57.140483 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.131.190:37467 (local address 127.3.131.129:53755)
0621 05:10:57.140664 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0621 05:10:57.140678 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0621 05:10:57.140736 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:10:57.140899 (+ 163us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:10:57.140900 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:10:57.141067 (+ 167us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:10:57.141116 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:10:57.141118 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:10:57.141898 (+ 780us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:10:57.141899 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:10:57.142559 (+ 660us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:10:57.142561 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:10:57.142936 (+ 375us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:10:57.142937 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:10:57.143001 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:10:57.143557 (+ 556us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 05:10:57.143561 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:10:57.145568 (+ 2007us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.131.190:37467: Server oryx/127.3.131.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":155,"thread_start_us":103,"threads_started":1}
W20260621 05:10:57.145740 12754 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 05:10:57.140582 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.131.129:53755 (local address 127.3.131.190:37467)
0621 05:10:57.140726 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0621 05:10:57.140728 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:10:57.140737 (+ 9us) server_negotiation.cc:407] Connection header received
0621 05:10:57.140783 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:10:57.140784 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:10:57.140793 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:10:57.140854 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:10:57.141253 (+ 399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:10:57.141803 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:10:57.142641 (+ 838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:10:57.142852 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:10:57.142935 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:10:57.145668 (+ 2733us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.131.129:53755: BlockingRecv error: recv got EOF from 127.3.131.129:53755 (error 108)
Metrics: {"server-negotiator.queue_time_us":115,"thread_start_us":89,"threads_started":1}
Jun 21 05:10:57 dist-test-slave-vrq1 krb5kdc[11752](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.131.130@KRBTEST.COM for oryx/127.3.131.190@KRBTEST.COM, Server not found in Kerberos database
Jun 21 05:10:57 dist-test-slave-vrq1 krb5kdc[11752](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.131.130@KRBTEST.COM for oryx/127.3.131.190@KRBTEST.COM, Server not found in Kerberos database
I20260621 05:10:57.292136 12755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:10:57.286540 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.131.190:37467 (local address 127.3.131.130:44713)
0621 05:10:57.286702 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0621 05:10:57.286715 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0621 05:10:57.286774 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:10:57.286953 (+ 179us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:10:57.286953 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:10:57.287116 (+ 163us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:10:57.287355 (+ 239us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:10:57.287358 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:10:57.288076 (+ 718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:10:57.288078 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:10:57.288927 (+ 849us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:10:57.288932 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:10:57.289386 (+ 454us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:10:57.289388 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:10:57.289452 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:10:57.289934 (+ 482us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 05:10:57.289940 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:10:57.292018 (+ 2078us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.131.190:37467: Server oryx/127.3.131.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":102,"threads_started":1}
W20260621 05:10:57.292196 12754 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 05:10:57.286737 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.131.130:44713 (local address 127.3.131.190:37467)
0621 05:10:57.286818 (+ 81us) server_negotiation.cc:206] Beginning negotiation
0621 05:10:57.286819 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0621 05:10:57.286829 (+ 10us) server_negotiation.cc:407] Connection header received
0621 05:10:57.286856 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:10:57.286857 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:10:57.286867 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:10:57.286932 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:10:57.287439 (+ 507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:10:57.287958 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:10:57.289036 (+ 1078us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:10:57.289289 (+ 253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:10:57.289385 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:10:57.292126 (+ 2741us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.131.130:44713: BlockingRecv error: recv got EOF from 127.3.131.130:44713 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
Jun 21 05:10:57 dist-test-slave-vrq1 krb5kdc[11752](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.131.131@KRBTEST.COM for oryx/127.3.131.190@KRBTEST.COM, Server not found in Kerberos database
Jun 21 05:10:57 dist-test-slave-vrq1 krb5kdc[11752](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.131.131@KRBTEST.COM for oryx/127.3.131.190@KRBTEST.COM, Server not found in Kerberos database
I20260621 05:10:57.446914 12756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:10:57.441954 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.131.190:37467 (local address 127.3.131.131:59033)
0621 05:10:57.442160 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0621 05:10:57.442174 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0621 05:10:57.442254 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:10:57.442468 (+ 214us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:10:57.442469 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:10:57.442637 (+ 168us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:10:57.442707 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:10:57.442709 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:10:57.443449 (+ 740us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:10:57.443450 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:10:57.444018 (+ 568us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:10:57.444020 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:10:57.444382 (+ 362us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:10:57.444383 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:10:57.444424 (+ 41us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:10:57.444960 (+ 536us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 05:10:57.444965 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:10:57.446841 (+ 1876us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.131.190:37467: Server oryx/127.3.131.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":182,"thread_start_us":106,"threads_started":1}
W20260621 05:10:57.447072 12754 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 05:10:57.442132 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.131.131:59033 (local address 127.3.131.190:37467)
0621 05:10:57.442222 (+ 90us) server_negotiation.cc:206] Beginning negotiation
0621 05:10:57.442224 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:10:57.442239 (+ 15us) server_negotiation.cc:407] Connection header received
0621 05:10:57.442347 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:10:57.442349 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:10:57.442366 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:10:57.442424 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:10:57.442819 (+ 395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:10:57.443370 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:10:57.444118 (+ 748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:10:57.444309 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:10:57.444373 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:10:57.446987 (+ 2614us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.131.131:59033: BlockingRecv error: recv got EOF from 127.3.131.131:59033 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
Jun 21 05:10:58 dist-test-slave-vrq1 krb5kdc[11752](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.131.129@KRBTEST.COM for oryx/127.3.131.190@KRBTEST.COM, Server not found in Kerberos database
Jun 21 05:10:58 dist-test-slave-vrq1 krb5kdc[11752](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.131.129@KRBTEST.COM for oryx/127.3.131.190@KRBTEST.COM, Server not found in Kerberos database
I20260621 05:10:58.151357 12757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:10:58.146244 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.131.190:37467 (local address 127.3.131.129:36959)
0621 05:10:58.146393 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0621 05:10:58.146404 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0621 05:10:58.146454 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:10:58.146640 (+ 186us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:10:58.146640 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:10:58.146802 (+ 162us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:10:58.146850 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:10:58.146852 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:10:58.147623 (+ 771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:10:58.147624 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:10:58.148161 (+ 537us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:10:58.148163 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:10:58.148567 (+ 404us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:10:58.148568 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:10:58.148615 (+ 47us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:10:58.149266 (+ 651us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 05:10:58.149271 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:10:58.151294 (+ 2023us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.131.190:37467: Server oryx/127.3.131.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":89,"threads_started":1}
W20260621 05:10:58.151569 12758 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 05:10:58.146317 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.131.129:36959 (local address 127.3.131.190:37467)
0621 05:10:58.146460 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0621 05:10:58.146462 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:10:58.146473 (+ 11us) server_negotiation.cc:407] Connection header received
0621 05:10:58.146506 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:10:58.146507 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:10:58.146544 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:10:58.146619 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:10:58.146948 (+ 329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:10:58.147529 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:10:58.148257 (+ 728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:10:58.148468 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:10:58.148530 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:10:58.151449 (+ 2919us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.131.129:36959: BlockingRecv error: recv got EOF from 127.3.131.129:36959 (error 108)
Metrics: {"server-negotiator.queue_time_us":121,"thread_start_us":87,"threads_started":1}
Jun 21 05:10:58 dist-test-slave-vrq1 krb5kdc[11752](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.131.130@KRBTEST.COM for oryx/127.3.131.190@KRBTEST.COM, Server not found in Kerberos database
Jun 21 05:10:58 dist-test-slave-vrq1 krb5kdc[11752](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.131.130@KRBTEST.COM for oryx/127.3.131.190@KRBTEST.COM, Server not found in Kerberos database
I20260621 05:10:58.297928 12759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:10:58.292807 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.131.190:37467 (local address 127.3.131.130:52109)
0621 05:10:58.292979 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0621 05:10:58.292995 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0621 05:10:58.293070 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:10:58.293229 (+ 159us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:10:58.293230 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:10:58.293392 (+ 162us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:10:58.293441 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:10:58.293443 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:10:58.294247 (+ 804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:10:58.294248 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:10:58.294808 (+ 560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:10:58.294810 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:10:58.295266 (+ 456us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:10:58.295267 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:10:58.295320 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:10:58.295853 (+ 533us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 05:10:58.295858 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:10:58.297833 (+ 1975us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.131.190:37467: Server oryx/127.3.131.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":72,"threads_started":1}
W20260621 05:10:58.298060 12758 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 05:10:58.292967 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.131.130:52109 (local address 127.3.131.190:37467)
0621 05:10:58.293070 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0621 05:10:58.293072 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:10:58.293083 (+ 11us) server_negotiation.cc:407] Connection header received
0621 05:10:58.293128 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:10:58.293129 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:10:58.293142 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:10:58.293209 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:10:58.293506 (+ 297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:10:58.294158 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:10:58.294894 (+ 736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:10:58.295123 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:10:58.295251 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:10:58.297939 (+ 2688us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.131.130:52109: BlockingRecv error: recv got EOF from 127.3.131.130:52109 (error 108)
Metrics: {"server-negotiator.queue_time_us":27}
Jun 21 05:10:58 dist-test-slave-vrq1 krb5kdc[11752](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.131.131@KRBTEST.COM for oryx/127.3.131.190@KRBTEST.COM, Server not found in Kerberos database
Jun 21 05:10:58 dist-test-slave-vrq1 krb5kdc[11752](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.131.131@KRBTEST.COM for oryx/127.3.131.190@KRBTEST.COM, Server not found in Kerberos database
I20260621 05:10:58.452435 12760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:10:58.447503 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.131.190:37467 (local address 127.3.131.131:39507)
0621 05:10:58.447636 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0621 05:10:58.447650 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0621 05:10:58.447715 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:10:58.447891 (+ 176us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:10:58.447892 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:10:58.448077 (+ 185us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:10:58.448123 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:10:58.448124 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:10:58.448791 (+ 667us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:10:58.448793 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:10:58.449346 (+ 553us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:10:58.449348 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:10:58.449731 (+ 383us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:10:58.449732 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:10:58.449822 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:10:58.450474 (+ 652us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 05:10:58.450479 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:10:58.452335 (+ 1856us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.131.190:37467: Server oryx/127.3.131.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":83,"threads_started":1}
W20260621 05:10:58.452484 12758 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 05:10:58.447606 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.131.131:39507 (local address 127.3.131.190:37467)
0621 05:10:58.447653 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0621 05:10:58.447655 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:10:58.447697 (+ 42us) server_negotiation.cc:407] Connection header received
0621 05:10:58.447789 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:10:58.447790 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:10:58.447804 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:10:58.447868 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:10:58.448204 (+ 336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:10:58.448702 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:10:58.449439 (+ 737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:10:58.449641 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:10:58.449731 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:10:58.452420 (+ 2689us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.131.131:39507: BlockingRecv error: recv got EOF from 127.3.131.131:39507 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
Jun 21 05:10:59 dist-test-slave-vrq1 krb5kdc[11752](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.131.129@KRBTEST.COM for oryx/127.3.131.190@KRBTEST.COM, Server not found in Kerberos database
Jun 21 05:10:59 dist-test-slave-vrq1 krb5kdc[11752](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.131.129@KRBTEST.COM for oryx/127.3.131.190@KRBTEST.COM, Server not found in Kerberos database
I20260621 05:10:59.157447 12761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:10:59.151995 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.131.190:37467 (local address 127.3.131.129:37919)
0621 05:10:59.152158 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0621 05:10:59.152173 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0621 05:10:59.152297 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:10:59.152525 (+ 228us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:10:59.152526 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:10:59.152723 (+ 197us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:10:59.152780 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:10:59.152782 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:10:59.153633 (+ 851us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:10:59.153634 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:10:59.154235 (+ 601us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:10:59.154237 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:10:59.154620 (+ 383us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:10:59.154621 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:10:59.154675 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:10:59.155291 (+ 616us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 05:10:59.155296 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:10:59.157384 (+ 2088us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.131.190:37467: Server oryx/127.3.131.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":74,"threads_started":1}
W20260621 05:10:59.157687 12762 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 05:10:59.152076 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.131.129:37919 (local address 127.3.131.190:37467)
0621 05:10:59.152295 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0621 05:10:59.152297 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:10:59.152308 (+ 11us) server_negotiation.cc:407] Connection header received
0621 05:10:59.152378 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:10:59.152379 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:10:59.152390 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:10:59.152446 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:10:59.152889 (+ 443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:10:59.153536 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:10:59.154328 (+ 792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:10:59.154525 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:10:59.154585 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:10:59.157594 (+ 3009us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.131.129:37919: BlockingRecv error: recv got EOF from 127.3.131.129:37919 (error 108)
Metrics: {"server-negotiator.queue_time_us":196,"thread_start_us":106,"threads_started":1}
Jun 21 05:10:59 dist-test-slave-vrq1 krb5kdc[11752](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.131.130@KRBTEST.COM for oryx/127.3.131.190@KRBTEST.COM, Server not found in Kerberos database
Jun 21 05:10:59 dist-test-slave-vrq1 krb5kdc[11752](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.131.130@KRBTEST.COM for oryx/127.3.131.190@KRBTEST.COM, Server not found in Kerberos database
I20260621 05:10:59.306133 12763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:10:59.298453 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.131.190:37467 (local address 127.3.131.130:36729)
0621 05:10:59.298577 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0621 05:10:59.298591 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0621 05:10:59.298677 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:10:59.299313 (+ 636us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:10:59.299314 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:10:59.299553 (+ 239us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:10:59.299638 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:10:59.299641 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:10:59.300752 (+ 1111us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:10:59.300753 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:10:59.301739 (+ 986us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:10:59.301742 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:10:59.302618 (+ 876us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:10:59.302619 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:10:59.302688 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:10:59.303871 (+ 1183us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 05:10:59.303877 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:10:59.306028 (+ 2151us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.131.190:37467: Server oryx/127.3.131.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":69,"threads_started":1}
W20260621 05:10:59.306198 12762 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 05:10:59.298547 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.131.130:36729 (local address 127.3.131.190:37467)
0621 05:10:59.298607 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0621 05:10:59.298609 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:10:59.298655 (+ 46us) server_negotiation.cc:407] Connection header received
0621 05:10:59.299149 (+ 494us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:10:59.299151 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:10:59.299163 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:10:59.299242 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:10:59.299750 (+ 508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:10:59.300620 (+ 870us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:10:59.301987 (+ 1367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:10:59.302350 (+ 363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:10:59.302617 (+ 267us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:10:59.306130 (+ 3513us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.131.130:36729: BlockingRecv error: recv got EOF from 127.3.131.130:36729 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
Jun 21 05:10:59 dist-test-slave-vrq1 krb5kdc[11752](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.131.131@KRBTEST.COM for oryx/127.3.131.190@KRBTEST.COM, Server not found in Kerberos database
Jun 21 05:10:59 dist-test-slave-vrq1 krb5kdc[11752](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.131.131@KRBTEST.COM for oryx/127.3.131.190@KRBTEST.COM, Server not found in Kerberos database
I20260621 05:10:59.458153 12764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:10:59.453092 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.131.190:37467 (local address 127.3.131.131:53851)
0621 05:10:59.453267 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0621 05:10:59.453282 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0621 05:10:59.453383 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:10:59.453592 (+ 209us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:10:59.453593 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:10:59.453786 (+ 193us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:10:59.453835 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:10:59.453837 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:10:59.454563 (+ 726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:10:59.454565 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:10:59.455103 (+ 538us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:10:59.455105 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:10:59.455543 (+ 438us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:10:59.455544 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:10:59.455614 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:10:59.456187 (+ 573us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 05:10:59.456191 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:10:59.458042 (+ 1851us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.131.190:37467: Server oryx/127.3.131.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":149,"thread_start_us":80,"threads_started":1}
W20260621 05:10:59.458206 12762 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 05:10:59.453191 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.131.131:53851 (local address 127.3.131.190:37467)
0621 05:10:59.453282 (+ 91us) server_negotiation.cc:206] Beginning negotiation
0621 05:10:59.453284 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:10:59.453362 (+ 78us) server_negotiation.cc:407] Connection header received
0621 05:10:59.453464 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:10:59.453464 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:10:59.453481 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:10:59.453585 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:10:59.453944 (+ 359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:10:59.454467 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:10:59.455220 (+ 753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:10:59.455444 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:10:59.455538 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:10:59.458140 (+ 2602us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.131.131:53851: BlockingRecv error: recv got EOF from 127.3.131.131:53851 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
Jun 21 05:11:00 dist-test-slave-vrq1 krb5kdc[11752](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.131.129@KRBTEST.COM for oryx/127.3.131.190@KRBTEST.COM, Server not found in Kerberos database
Jun 21 05:11:00 dist-test-slave-vrq1 krb5kdc[11752](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.131.129@KRBTEST.COM for oryx/127.3.131.190@KRBTEST.COM, Server not found in Kerberos database
I20260621 05:11:00.167800 12765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:11:00.157986 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.131.190:37467 (local address 127.3.131.129:47213)
0621 05:11:00.158164 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0621 05:11:00.158175 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0621 05:11:00.158229 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:11:00.158371 (+ 142us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:11:00.158372 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:11:00.158534 (+ 162us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:11:00.158614 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:11:00.158615 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:11:00.159313 (+ 698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:11:00.159314 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:11:00.159867 (+ 553us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:11:00.159869 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:11:00.160254 (+ 385us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:11:00.160254 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:11:00.160297 (+ 43us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:11:00.165490 (+ 5193us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 05:11:00.165497 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:11:00.167683 (+ 2186us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.131.190:37467: Server oryx/127.3.131.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":158,"thread_start_us":109,"threads_started":1}
W20260621 05:11:00.167902 12766 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 05:11:00.158105 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.131.129:47213 (local address 127.3.131.190:37467)
0621 05:11:00.158235 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0621 05:11:00.158237 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:11:00.158247 (+ 10us) server_negotiation.cc:407] Connection header received
0621 05:11:00.158277 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:11:00.158278 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:11:00.158290 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:11:00.158354 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:11:00.158681 (+ 327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:11:00.159246 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:11:00.159947 (+ 701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:11:00.160182 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:11:00.160243 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:11:00.167820 (+ 7577us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.131.129:47213: BlockingRecv error: recv got EOF from 127.3.131.129:47213 (error 108)
Metrics: {"server-negotiator.queue_time_us":105,"thread_start_us":78,"threads_started":1}
Jun 21 05:11:00 dist-test-slave-vrq1 krb5kdc[11752](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.131.130@KRBTEST.COM for oryx/127.3.131.190@KRBTEST.COM, Server not found in Kerberos database
Jun 21 05:11:00 dist-test-slave-vrq1 krb5kdc[11752](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.131.130@KRBTEST.COM for oryx/127.3.131.190@KRBTEST.COM, Server not found in Kerberos database
I20260621 05:11:00.312110 12767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:11:00.306729 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.131.190:37467 (local address 127.3.131.130:37637)
0621 05:11:00.306949 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0621 05:11:00.306962 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0621 05:11:00.307087 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:11:00.307297 (+ 210us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:11:00.307298 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:11:00.307559 (+ 261us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:11:00.307608 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:11:00.307610 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:11:00.308349 (+ 739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:11:00.308350 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:11:00.308908 (+ 558us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:11:00.308909 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:11:00.309296 (+ 387us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:11:00.309297 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:11:00.309343 (+ 46us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:11:00.309851 (+ 508us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 05:11:00.309855 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:11:00.312023 (+ 2168us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.131.190:37467: Server oryx/127.3.131.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":200,"thread_start_us":174,"threads_started":1}
W20260621 05:11:00.312300 12766 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 05:11:00.306872 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.131.130:37637 (local address 127.3.131.190:37467)
0621 05:11:00.306946 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0621 05:11:00.306949 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0621 05:11:00.307085 (+ 136us) server_negotiation.cc:407] Connection header received
0621 05:11:00.307163 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:11:00.307163 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:11:00.307195 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:11:00.307252 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:11:00.307708 (+ 456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:11:00.308249 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:11:00.309008 (+ 759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:11:00.309211 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:11:00.309274 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:11:00.312238 (+ 2964us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.131.130:37637: BlockingRecv error: recv got EOF from 127.3.131.130:37637 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
Jun 21 05:11:00 dist-test-slave-vrq1 krb5kdc[11752](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.131.131@KRBTEST.COM for oryx/127.3.131.190@KRBTEST.COM, Server not found in Kerberos database
Jun 21 05:11:00 dist-test-slave-vrq1 krb5kdc[11752](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.131.131@KRBTEST.COM for oryx/127.3.131.190@KRBTEST.COM, Server not found in Kerberos database
I20260621 05:11:00.463714 12768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:11:00.458716 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.131.190:37467 (local address 127.3.131.131:50985)
0621 05:11:00.458893 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0621 05:11:00.458904 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0621 05:11:00.458972 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:11:00.459135 (+ 163us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:11:00.459136 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:11:00.459321 (+ 185us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:11:00.459373 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:11:00.459375 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:11:00.460081 (+ 706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:11:00.460082 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:11:00.460632 (+ 550us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:11:00.460633 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:11:00.461062 (+ 429us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:11:00.461063 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:11:00.461117 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:11:00.461667 (+ 550us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 05:11:00.461671 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:11:00.463652 (+ 1981us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.131.190:37467: Server oryx/127.3.131.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":149,"thread_start_us":75,"threads_started":1}
W20260621 05:11:00.463865 12766 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 05:11:00.458828 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.131.131:50985 (local address 127.3.131.190:37467)
0621 05:11:00.458881 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0621 05:11:00.458883 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:11:00.458970 (+ 87us) server_negotiation.cc:407] Connection header received
0621 05:11:00.459017 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:11:00.459018 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:11:00.459030 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:11:00.459077 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:11:00.459453 (+ 376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:11:00.459956 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:11:00.460734 (+ 778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:11:00.460930 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:11:00.460995 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:11:00.463771 (+ 2776us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.131.131:50985: BlockingRecv error: recv got EOF from 127.3.131.131:50985 (error 108)
Metrics: {"server-negotiator.queue_time_us":28}
Jun 21 05:11:01 dist-test-slave-vrq1 krb5kdc[11752](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.131.129@KRBTEST.COM for oryx/127.3.131.190@KRBTEST.COM, Server not found in Kerberos database
Jun 21 05:11:01 dist-test-slave-vrq1 krb5kdc[11752](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.131.129@KRBTEST.COM for oryx/127.3.131.190@KRBTEST.COM, Server not found in Kerberos database
I20260621 05:11:01.173403 12769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:11:01.168363 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.131.190:37467 (local address 127.3.131.129:52657)
0621 05:11:01.168518 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0621 05:11:01.168531 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0621 05:11:01.168638 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:11:01.168850 (+ 212us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:11:01.168851 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:11:01.169063 (+ 212us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:11:01.169112 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:11:01.169114 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:11:01.169905 (+ 791us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:11:01.169907 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:11:01.170486 (+ 579us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:11:01.170489 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:11:01.170923 (+ 434us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:11:01.170924 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:11:01.170983 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:11:01.171569 (+ 586us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 05:11:01.171574 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:11:01.173342 (+ 1768us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.131.190:37467: Server oryx/127.3.131.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":100,"threads_started":1}
W20260621 05:11:01.173601 12770 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 05:11:01.168489 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.131.129:52657 (local address 127.3.131.190:37467)
0621 05:11:01.168642 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0621 05:11:01.168644 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:11:01.168653 (+ 9us) server_negotiation.cc:407] Connection header received
0621 05:11:01.168743 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:11:01.168744 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:11:01.168754 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:11:01.168823 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:11:01.169237 (+ 414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:11:01.169796 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:11:01.170591 (+ 795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:11:01.170810 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:11:01.170873 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:11:01.173471 (+ 2598us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.131.129:52657: BlockingRecv error: recv got EOF from 127.3.131.129:52657 (error 108)
Metrics: {"server-negotiator.queue_time_us":127,"thread_start_us":99,"threads_started":1}
Jun 21 05:11:01 dist-test-slave-vrq1 krb5kdc[11752](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.131.130@KRBTEST.COM for oryx/127.3.131.190@KRBTEST.COM, Server not found in Kerberos database
Jun 21 05:11:01 dist-test-slave-vrq1 krb5kdc[11752](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.131.130@KRBTEST.COM for oryx/127.3.131.190@KRBTEST.COM, Server not found in Kerberos database
I20260621 05:11:01.317695 12771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:11:01.312625 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.131.190:37467 (local address 127.3.131.130:42157)
0621 05:11:01.312773 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0621 05:11:01.312784 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0621 05:11:01.312836 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:11:01.312979 (+ 143us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:11:01.312980 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:11:01.313198 (+ 218us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:11:01.313248 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:11:01.313250 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:11:01.314039 (+ 789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:11:01.314040 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:11:01.314609 (+ 569us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:11:01.314611 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:11:01.315044 (+ 433us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:11:01.315045 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:11:01.315089 (+ 44us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:11:01.315733 (+ 644us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 05:11:01.315738 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:11:01.317606 (+ 1868us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.131.190:37467: Server oryx/127.3.131.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":69,"threads_started":1}
W20260621 05:11:01.317759 12770 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 05:11:01.312731 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.131.130:42157 (local address 127.3.131.190:37467)
0621 05:11:01.312819 (+ 88us) server_negotiation.cc:206] Beginning negotiation
0621 05:11:01.312821 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:11:01.312840 (+ 19us) server_negotiation.cc:407] Connection header received
0621 05:11:01.312902 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:11:01.312903 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:11:01.312914 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:11:01.312955 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:11:01.313358 (+ 403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:11:01.313896 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:11:01.314730 (+ 834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:11:01.314929 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:11:01.315036 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:11:01.317709 (+ 2673us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.131.130:42157: BlockingRecv error: recv got EOF from 127.3.131.130:42157 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
Jun 21 05:11:01 dist-test-slave-vrq1 krb5kdc[11752](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.131.131@KRBTEST.COM for oryx/127.3.131.190@KRBTEST.COM, Server not found in Kerberos database
Jun 21 05:11:01 dist-test-slave-vrq1 krb5kdc[11752](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.131.131@KRBTEST.COM for oryx/127.3.131.190@KRBTEST.COM, Server not found in Kerberos database
I20260621 05:11:01.469828 12772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:11:01.464394 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.131.190:37467 (local address 127.3.131.131:49175)
0621 05:11:01.464545 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0621 05:11:01.464556 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0621 05:11:01.464652 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:11:01.464820 (+ 168us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:11:01.464821 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:11:01.465003 (+ 182us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:11:01.465088 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:11:01.465090 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:11:01.465793 (+ 703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:11:01.465794 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:11:01.466467 (+ 673us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:11:01.466470 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:11:01.466885 (+ 415us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:11:01.466886 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:11:01.466945 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:11:01.467455 (+ 510us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 05:11:01.467460 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:11:01.469766 (+ 2306us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.131.190:37467: Server oryx/127.3.131.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":60,"threads_started":1}
W20260621 05:11:01.470029 12770 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 05:11:01.464539 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.131.131:49175 (local address 127.3.131.190:37467)
0621 05:11:01.464657 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0621 05:11:01.464660 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0621 05:11:01.464691 (+ 31us) server_negotiation.cc:407] Connection header received
0621 05:11:01.464724 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:11:01.464731 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:11:01.464744 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:11:01.464812 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:11:01.465180 (+ 368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:11:01.465708 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:11:01.466591 (+ 883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:11:01.466805 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:11:01.466887 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:11:01.469962 (+ 3075us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.131.131:49175: BlockingRecv error: recv got EOF from 127.3.131.131:49175 (error 108)
Metrics: {"server-negotiator.queue_time_us":90}
Jun 21 05:11:02 dist-test-slave-vrq1 krb5kdc[11752](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.131.129@KRBTEST.COM for oryx/127.3.131.190@KRBTEST.COM, Server not found in Kerberos database
Jun 21 05:11:02 dist-test-slave-vrq1 krb5kdc[11752](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.131.129@KRBTEST.COM for oryx/127.3.131.190@KRBTEST.COM, Server not found in Kerberos database
I20260621 05:11:02.179040 12773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:11:02.174026 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.131.190:37467 (local address 127.3.131.129:45985)
0621 05:11:02.174155 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0621 05:11:02.174167 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0621 05:11:02.174229 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:11:02.174425 (+ 196us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:11:02.174427 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:11:02.174596 (+ 169us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:11:02.174656 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:11:02.174657 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:11:02.175397 (+ 740us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:11:02.175398 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:11:02.175958 (+ 560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:11:02.175960 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:11:02.176380 (+ 420us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:11:02.176381 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:11:02.176430 (+ 49us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:11:02.176998 (+ 568us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 05:11:02.177004 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:11:02.178979 (+ 1975us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.131.190:37467: Server oryx/127.3.131.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":76,"threads_started":1}
W20260621 05:11:02.179272 12774 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 05:11:02.174069 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.131.129:45985 (local address 127.3.131.190:37467)
0621 05:11:02.174257 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0621 05:11:02.174259 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:11:02.174268 (+ 9us) server_negotiation.cc:407] Connection header received
0621 05:11:02.174302 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:11:02.174302 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:11:02.174313 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:11:02.174378 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:11:02.174758 (+ 380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:11:02.175319 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:11:02.176087 (+ 768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:11:02.176286 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:11:02.176353 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:11:02.179149 (+ 2796us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.131.129:45985: BlockingRecv error: recv got EOF from 127.3.131.129:45985 (error 108)
Metrics: {"server-negotiator.queue_time_us":166,"thread_start_us":93,"threads_started":1}
Jun 21 05:11:02 dist-test-slave-vrq1 krb5kdc[11752](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.131.130@KRBTEST.COM for oryx/127.3.131.190@KRBTEST.COM, Server not found in Kerberos database
Jun 21 05:11:02 dist-test-slave-vrq1 krb5kdc[11752](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.131.130@KRBTEST.COM for oryx/127.3.131.190@KRBTEST.COM, Server not found in Kerberos database
I20260621 05:11:02.323354 12775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:11:02.318301 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.131.190:37467 (local address 127.3.131.130:35011)
0621 05:11:02.318458 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0621 05:11:02.318472 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0621 05:11:02.318551 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:11:02.318720 (+ 169us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:11:02.318721 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:11:02.318917 (+ 196us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:11:02.318969 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:11:02.318972 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:11:02.319708 (+ 736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:11:02.319709 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:11:02.320249 (+ 540us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:11:02.320251 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:11:02.320607 (+ 356us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:11:02.320609 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:11:02.320656 (+ 47us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:11:02.321156 (+ 500us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 05:11:02.321161 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:11:02.323290 (+ 2129us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.131.190:37467: Server oryx/127.3.131.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":97,"threads_started":1}
W20260621 05:11:02.323572 12774 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 05:11:02.318416 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.131.130:35011 (local address 127.3.131.190:37467)
0621 05:11:02.318472 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0621 05:11:02.318474 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:11:02.318536 (+ 62us) server_negotiation.cc:407] Connection header received
0621 05:11:02.318602 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:11:02.318602 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:11:02.318613 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:11:02.318678 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:11:02.319063 (+ 385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:11:02.319623 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:11:02.320340 (+ 717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:11:02.320534 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:11:02.320596 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:11:02.323511 (+ 2915us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.131.130:35011: BlockingRecv error: recv got EOF from 127.3.131.130:35011 (error 108)
Metrics: {"server-negotiator.queue_time_us":26}
Jun 21 05:11:02 dist-test-slave-vrq1 krb5kdc[11752](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.131.131@KRBTEST.COM for oryx/127.3.131.190@KRBTEST.COM, Server not found in Kerberos database
Jun 21 05:11:02 dist-test-slave-vrq1 krb5kdc[11752](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.131.131@KRBTEST.COM for oryx/127.3.131.190@KRBTEST.COM, Server not found in Kerberos database
I20260621 05:11:02.475878 12776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:11:02.470477 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.131.190:37467 (local address 127.3.131.131:46607)
0621 05:11:02.470657 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0621 05:11:02.470688 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0621 05:11:02.470776 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:11:02.470965 (+ 189us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:11:02.470966 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:11:02.471158 (+ 192us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:11:02.471232 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:11:02.471234 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:11:02.471936 (+ 702us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:11:02.471937 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:11:02.472487 (+ 550us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:11:02.472489 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:11:02.473000 (+ 511us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:11:02.473001 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:11:02.473083 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:11:02.473654 (+ 571us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 05:11:02.473660 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:11:02.475815 (+ 2155us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.131.190:37467: Server oryx/127.3.131.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":159,"thread_start_us":115,"threads_started":1}
W20260621 05:11:02.476091 12774 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 05:11:02.470643 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.131.131:46607 (local address 127.3.131.190:37467)
0621 05:11:02.470719 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0621 05:11:02.470721 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:11:02.470756 (+ 35us) server_negotiation.cc:407] Connection header received
0621 05:11:02.470871 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:11:02.470871 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:11:02.470890 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:11:02.470959 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:11:02.471320 (+ 361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:11:02.471846 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:11:02.472593 (+ 747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:11:02.472790 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:11:02.472997 (+ 207us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:11:02.476011 (+ 3014us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.131.131:46607: BlockingRecv error: recv got EOF from 127.3.131.131:46607 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
Jun 21 05:11:03 dist-test-slave-vrq1 krb5kdc[11752](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.131.129@KRBTEST.COM for oryx/127.3.131.190@KRBTEST.COM, Server not found in Kerberos database
Jun 21 05:11:03 dist-test-slave-vrq1 krb5kdc[11752](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.131.129@KRBTEST.COM for oryx/127.3.131.190@KRBTEST.COM, Server not found in Kerberos database
I20260621 05:11:03.184675 12777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:11:03.179610 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.131.190:37467 (local address 127.3.131.129:41125)
0621 05:11:03.179794 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0621 05:11:03.179808 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0621 05:11:03.179889 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:11:03.180100 (+ 211us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:11:03.180101 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:11:03.180262 (+ 161us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:11:03.180310 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:11:03.180312 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:11:03.180998 (+ 686us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:11:03.180999 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:11:03.181715 (+ 716us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:11:03.181718 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:11:03.182135 (+ 417us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:11:03.182136 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:11:03.182185 (+ 49us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:11:03.182706 (+ 521us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 05:11:03.182710 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:11:03.184609 (+ 1899us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.131.190:37467: Server oryx/127.3.131.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":80,"threads_started":1}
W20260621 05:11:03.184834 12778 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 05:11:03.179709 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.131.129:41125 (local address 127.3.131.190:37467)
0621 05:11:03.179818 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0621 05:11:03.179820 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:11:03.179864 (+ 44us) server_negotiation.cc:407] Connection header received
0621 05:11:03.179936 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:11:03.179936 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:11:03.179980 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:11:03.180040 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:11:03.180403 (+ 363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:11:03.180906 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:11:03.181800 (+ 894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:11:03.182040 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:11:03.182100 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:11:03.184732 (+ 2632us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.131.129:41125: BlockingRecv error: recv got EOF from 127.3.131.129:41125 (error 108)
Metrics: {"server-negotiator.queue_time_us":90,"thread_start_us":66,"threads_started":1}
Jun 21 05:11:03 dist-test-slave-vrq1 krb5kdc[11752](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.131.130@KRBTEST.COM for oryx/127.3.131.190@KRBTEST.COM, Server not found in Kerberos database
Jun 21 05:11:03 dist-test-slave-vrq1 krb5kdc[11752](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.131.130@KRBTEST.COM for oryx/127.3.131.190@KRBTEST.COM, Server not found in Kerberos database
I20260621 05:11:03.329018 12779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:11:03.323911 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.131.190:37467 (local address 127.3.131.130:60397)
0621 05:11:03.324061 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0621 05:11:03.324073 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0621 05:11:03.324163 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:11:03.324342 (+ 179us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:11:03.324343 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:11:03.324534 (+ 191us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:11:03.324591 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:11:03.324594 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:11:03.325311 (+ 717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:11:03.325313 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:11:03.325862 (+ 549us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:11:03.325864 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:11:03.326310 (+ 446us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:11:03.326311 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:11:03.326373 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:11:03.326848 (+ 475us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 05:11:03.326852 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:11:03.328946 (+ 2094us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.131.190:37467: Server oryx/127.3.131.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":97,"threads_started":1}
W20260621 05:11:03.329156 12778 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 05:11:03.323957 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.131.130:60397 (local address 127.3.131.190:37467)
0621 05:11:03.324003 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0621 05:11:03.324005 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:11:03.324150 (+ 145us) server_negotiation.cc:407] Connection header received
0621 05:11:03.324244 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:11:03.324245 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:11:03.324259 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:11:03.324314 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:11:03.324698 (+ 384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:11:03.325216 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:11:03.325982 (+ 766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:11:03.326211 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:11:03.326307 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:11:03.329074 (+ 2767us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.131.130:60397: BlockingRecv error: recv got EOF from 127.3.131.130:60397 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
Jun 21 05:11:03 dist-test-slave-vrq1 krb5kdc[11752](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.131.131@KRBTEST.COM for oryx/127.3.131.190@KRBTEST.COM, Server not found in Kerberos database
Jun 21 05:11:03 dist-test-slave-vrq1 krb5kdc[11752](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.131.131@KRBTEST.COM for oryx/127.3.131.190@KRBTEST.COM, Server not found in Kerberos database
I20260621 05:11:03.481902 12780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:11:03.476428 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.131.190:37467 (local address 127.3.131.131:51071)
0621 05:11:03.476593 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0621 05:11:03.476612 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0621 05:11:03.476674 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:11:03.476848 (+ 174us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:11:03.476849 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:11:03.477012 (+ 163us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:11:03.477069 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:11:03.477071 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:11:03.477916 (+ 845us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:11:03.477918 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:11:03.478722 (+ 804us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:11:03.478725 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:11:03.479095 (+ 370us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:11:03.479096 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:11:03.479139 (+ 43us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:11:03.479715 (+ 576us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 05:11:03.479720 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:11:03.481841 (+ 2121us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.131.190:37467: Server oryx/127.3.131.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":112,"threads_started":1}
W20260621 05:11:03.482105 12778 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 05:11:03.476534 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.131.131:51071 (local address 127.3.131.190:37467)
0621 05:11:03.476623 (+ 89us) server_negotiation.cc:206] Beginning negotiation
0621 05:11:03.476626 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0621 05:11:03.476662 (+ 36us) server_negotiation.cc:407] Connection header received
0621 05:11:03.476741 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:11:03.476741 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:11:03.476756 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:11:03.476822 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:11:03.477201 (+ 379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:11:03.477832 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:11:03.478796 (+ 964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:11:03.478989 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:11:03.479053 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:11:03.481960 (+ 2907us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.131.131:51071: BlockingRecv error: recv got EOF from 127.3.131.131:51071 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
Jun 21 05:11:04 dist-test-slave-vrq1 krb5kdc[11752](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.131.129@KRBTEST.COM for oryx/127.3.131.190@KRBTEST.COM, Server not found in Kerberos database
Jun 21 05:11:04 dist-test-slave-vrq1 krb5kdc[11752](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.131.129@KRBTEST.COM for oryx/127.3.131.190@KRBTEST.COM, Server not found in Kerberos database
I20260621 05:11:04.190837 12781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:11:04.185318 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.131.190:37467 (local address 127.3.131.129:35473)
0621 05:11:04.185475 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0621 05:11:04.185486 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0621 05:11:04.185540 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:11:04.185785 (+ 245us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:11:04.185786 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:11:04.185956 (+ 170us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:11:04.186008 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:11:04.186010 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:11:04.186777 (+ 767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:11:04.186778 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:11:04.187490 (+ 712us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:11:04.187493 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:11:04.187866 (+ 373us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:11:04.187866 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:11:04.187911 (+ 45us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:11:04.188592 (+ 681us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 05:11:04.188598 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:11:04.190778 (+ 2180us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.131.190:37467: Server oryx/127.3.131.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":90,"threads_started":1}
W20260621 05:11:04.191059 12782 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 05:11:04.185478 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.131.129:35473 (local address 127.3.131.190:37467)
0621 05:11:04.185622 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0621 05:11:04.185623 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0621 05:11:04.185632 (+ 9us) server_negotiation.cc:407] Connection header received
0621 05:11:04.185677 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:11:04.185677 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:11:04.185687 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:11:04.185757 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:11:04.186119 (+ 362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:11:04.186676 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:11:04.187585 (+ 909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:11:04.187790 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:11:04.187849 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:11:04.190943 (+ 3094us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.131.129:35473: BlockingRecv error: recv got EOF from 127.3.131.129:35473 (error 108)
Metrics: {"server-negotiator.queue_time_us":115,"thread_start_us":79,"threads_started":1}
Jun 21 05:11:04 dist-test-slave-vrq1 krb5kdc[11752](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.131.130@KRBTEST.COM for oryx/127.3.131.190@KRBTEST.COM, Server not found in Kerberos database
Jun 21 05:11:04 dist-test-slave-vrq1 krb5kdc[11752](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.131.130@KRBTEST.COM for oryx/127.3.131.190@KRBTEST.COM, Server not found in Kerberos database
I20260621 05:11:04.334949 12783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:11:04.329665 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.131.190:37467 (local address 127.3.131.130:42937)
0621 05:11:04.329834 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0621 05:11:04.329851 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0621 05:11:04.329904 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:11:04.330077 (+ 173us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:11:04.330078 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:11:04.330248 (+ 170us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:11:04.330298 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:11:04.330300 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:11:04.331082 (+ 782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:11:04.331083 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:11:04.331656 (+ 573us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:11:04.331657 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:11:04.332019 (+ 362us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:11:04.332020 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:11:04.332058 (+ 38us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:11:04.332528 (+ 470us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 05:11:04.332533 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:11:04.334850 (+ 2317us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.131.190:37467: Server oryx/127.3.131.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":104,"threads_started":1}
W20260621 05:11:04.334993 12782 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 05:11:04.329799 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.131.130:42937 (local address 127.3.131.190:37467)
0621 05:11:04.329876 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0621 05:11:04.329878 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:11:04.329893 (+ 15us) server_negotiation.cc:407] Connection header received
0621 05:11:04.329975 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:11:04.329976 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:11:04.329986 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:11:04.330054 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:11:04.330376 (+ 322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:11:04.331007 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:11:04.331745 (+ 738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:11:04.331947 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:11:04.332006 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:11:04.334931 (+ 2925us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.131.130:42937: BlockingRecv error: recv got EOF from 127.3.131.130:42937 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
Jun 21 05:11:04 dist-test-slave-vrq1 krb5kdc[11752](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.131.131@KRBTEST.COM for oryx/127.3.131.190@KRBTEST.COM, Server not found in Kerberos database
Jun 21 05:11:04 dist-test-slave-vrq1 krb5kdc[11752](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.131.131@KRBTEST.COM for oryx/127.3.131.190@KRBTEST.COM, Server not found in Kerberos database
I20260621 05:11:04.487344 12784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:11:04.482526 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.131.190:37467 (local address 127.3.131.131:56735)
0621 05:11:04.482703 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0621 05:11:04.482717 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0621 05:11:04.482810 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:11:04.482987 (+ 177us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:11:04.482988 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:11:04.483164 (+ 176us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:11:04.483240 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:11:04.483242 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:11:04.483933 (+ 691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:11:04.483934 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:11:04.484509 (+ 575us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:11:04.484511 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:11:04.484897 (+ 386us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:11:04.484897 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:11:04.484962 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:11:04.485427 (+ 465us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 05:11:04.485431 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:11:04.487282 (+ 1851us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.131.190:37467: Server oryx/127.3.131.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":160,"thread_start_us":91,"threads_started":1}
W20260621 05:11:04.487465 12782 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 05:11:04.482624 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.131.131:56735 (local address 127.3.131.190:37467)
0621 05:11:04.482708 (+ 84us) server_negotiation.cc:206] Beginning negotiation
0621 05:11:04.482710 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:11:04.482791 (+ 81us) server_negotiation.cc:407] Connection header received
0621 05:11:04.482897 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:11:04.482898 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:11:04.482911 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:11:04.482973 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:11:04.483322 (+ 349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:11:04.483835 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:11:04.484604 (+ 769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:11:04.484811 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:11:04.484895 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:11:04.487383 (+ 2488us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.131.131:56735: BlockingRecv error: recv got EOF from 127.3.131.131:56735 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
Jun 21 05:11:05 dist-test-slave-vrq1 krb5kdc[11752](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.131.129@KRBTEST.COM for oryx/127.3.131.190@KRBTEST.COM, Server not found in Kerberos database
Jun 21 05:11:05 dist-test-slave-vrq1 krb5kdc[11752](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.131.129@KRBTEST.COM for oryx/127.3.131.190@KRBTEST.COM, Server not found in Kerberos database
I20260621 05:11:05.199694 12785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:11:05.191477 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.131.190:37467 (local address 127.3.131.129:39477)
0621 05:11:05.191644 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0621 05:11:05.191663 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0621 05:11:05.191727 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:11:05.191940 (+ 213us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:11:05.191941 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:11:05.192153 (+ 212us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:11:05.192202 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:11:05.192203 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:11:05.192965 (+ 762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:11:05.192966 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:11:05.193509 (+ 543us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:11:05.193510 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:11:05.193913 (+ 403us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:11:05.193914 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:11:05.193965 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:11:05.197676 (+ 3711us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 05:11:05.197682 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:11:05.199616 (+ 1934us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.131.190:37467: Server oryx/127.3.131.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":115,"threads_started":1}
W20260621 05:11:05.199831 12786 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 05:11:05.191622 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.131.129:39477 (local address 127.3.131.190:37467)
0621 05:11:05.191789 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0621 05:11:05.191791 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:11:05.191801 (+ 10us) server_negotiation.cc:407] Connection header received
0621 05:11:05.191828 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:11:05.191829 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:11:05.191840 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:11:05.191918 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:11:05.192280 (+ 362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:11:05.192852 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:11:05.193607 (+ 755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:11:05.193828 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:11:05.193893 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:11:05.199732 (+ 5839us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.131.129:39477: BlockingRecv error: recv got EOF from 127.3.131.129:39477 (error 108)
Metrics: {"server-negotiator.queue_time_us":140,"thread_start_us":107,"threads_started":1}
Jun 21 05:11:05 dist-test-slave-vrq1 krb5kdc[11752](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.131.130@KRBTEST.COM for oryx/127.3.131.190@KRBTEST.COM, Server not found in Kerberos database
Jun 21 05:11:05 dist-test-slave-vrq1 krb5kdc[11752](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.131.130@KRBTEST.COM for oryx/127.3.131.190@KRBTEST.COM, Server not found in Kerberos database
I20260621 05:11:05.340736 12787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:11:05.335561 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.131.190:37467 (local address 127.3.131.130:47859)
0621 05:11:05.335743 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0621 05:11:05.335775 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0621 05:11:05.335857 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:11:05.336022 (+ 165us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:11:05.336023 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:11:05.336209 (+ 186us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:11:05.336257 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:11:05.336259 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:11:05.336958 (+ 699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:11:05.336959 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:11:05.337532 (+ 573us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:11:05.337534 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:11:05.337910 (+ 376us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:11:05.337911 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:11:05.337970 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:11:05.338486 (+ 516us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 05:11:05.338491 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:11:05.340607 (+ 2116us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.131.190:37467: Server oryx/127.3.131.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":162,"thread_start_us":117,"threads_started":1}
W20260621 05:11:05.340765 12786 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 05:11:05.335731 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.131.130:47859 (local address 127.3.131.190:37467)
0621 05:11:05.335802 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0621 05:11:05.335804 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:11:05.335834 (+ 30us) server_negotiation.cc:407] Connection header received
0621 05:11:05.335927 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:11:05.335929 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:11:05.335942 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:11:05.336015 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:11:05.336341 (+ 326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:11:05.336856 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:11:05.337631 (+ 775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:11:05.337832 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:11:05.337912 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:11:05.340702 (+ 2790us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.131.130:47859: BlockingRecv error: recv got EOF from 127.3.131.130:47859 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
Jun 21 05:11:05 dist-test-slave-vrq1 krb5kdc[11752](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.131.131@KRBTEST.COM for oryx/127.3.131.190@KRBTEST.COM, Server not found in Kerberos database
Jun 21 05:11:05 dist-test-slave-vrq1 krb5kdc[11752](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.131.131@KRBTEST.COM for oryx/127.3.131.190@KRBTEST.COM, Server not found in Kerberos database
I20260621 05:11:05.493399 12788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:11:05.487879 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.131.190:37467 (local address 127.3.131.131:43341)
0621 05:11:05.488100 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0621 05:11:05.488116 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0621 05:11:05.488177 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:11:05.488317 (+ 140us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:11:05.488317 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:11:05.488553 (+ 236us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:11:05.488601 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:11:05.488603 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:11:05.489466 (+ 863us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:11:05.489467 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:11:05.489997 (+ 530us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:11:05.489999 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:11:05.490439 (+ 440us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:11:05.490441 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:11:05.490499 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:11:05.491156 (+ 657us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 05:11:05.491161 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:11:05.493320 (+ 2159us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.131.190:37467: Server oryx/127.3.131.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":196,"thread_start_us":156,"threads_started":1}
W20260621 05:11:05.493666 12786 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 05:11:05.488052 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.131.131:43341 (local address 127.3.131.190:37467)
0621 05:11:05.488108 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0621 05:11:05.488110 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:11:05.488163 (+ 53us) server_negotiation.cc:407] Connection header received
0621 05:11:05.488225 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:11:05.488225 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:11:05.488237 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:11:05.488295 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:11:05.488723 (+ 428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:11:05.489332 (+ 609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:11:05.490136 (+ 804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:11:05.490340 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:11:05.490402 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:11:05.493598 (+ 3196us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.131.131:43341: BlockingRecv error: recv got EOF from 127.3.131.131:43341 (error 108)
Metrics: {"server-negotiator.queue_time_us":24}
Jun 21 05:11:06 dist-test-slave-vrq1 krb5kdc[11752](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.131.129@KRBTEST.COM for oryx/127.3.131.190@KRBTEST.COM, Server not found in Kerberos database
Jun 21 05:11:06 dist-test-slave-vrq1 krb5kdc[11752](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.131.129@KRBTEST.COM for oryx/127.3.131.190@KRBTEST.COM, Server not found in Kerberos database
I20260621 05:11:06.207099 12789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:11:06.200423 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.131.190:37467 (local address 127.3.131.129:46769)
0621 05:11:06.200558 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0621 05:11:06.200575 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0621 05:11:06.200692 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:11:06.200985 (+ 293us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:11:06.200986 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:11:06.201166 (+ 180us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:11:06.201220 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:11:06.201221 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:11:06.202198 (+ 977us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:11:06.202199 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:11:06.202991 (+ 792us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:11:06.202995 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:11:06.203547 (+ 552us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:11:06.203548 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:11:06.203601 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:11:06.204262 (+ 661us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 05:11:06.204267 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:11:06.207036 (+ 2769us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.131.190:37467: Server oryx/127.3.131.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":76,"threads_started":1}
W20260621 05:11:06.207412 12790 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 05:11:06.200538 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.131.129:46769 (local address 127.3.131.190:37467)
0621 05:11:06.200831 (+ 293us) server_negotiation.cc:206] Beginning negotiation
0621 05:11:06.200833 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:11:06.200844 (+ 11us) server_negotiation.cc:407] Connection header received
0621 05:11:06.200871 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:11:06.200872 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:11:06.200882 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:11:06.200957 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:11:06.201282 (+ 325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:11:06.202096 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:11:06.203134 (+ 1038us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:11:06.203375 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:11:06.203446 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:11:06.207300 (+ 3854us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.131.129:46769: BlockingRecv error: recv got EOF from 127.3.131.129:46769 (error 108)
Metrics: {"server-negotiator.queue_time_us":271,"thread_start_us":195,"threads_started":1}
Jun 21 05:11:06 dist-test-slave-vrq1 krb5kdc[11752](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.131.130@KRBTEST.COM for oryx/127.3.131.190@KRBTEST.COM, Server not found in Kerberos database
Jun 21 05:11:06 dist-test-slave-vrq1 krb5kdc[11752](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.131.130@KRBTEST.COM for oryx/127.3.131.190@KRBTEST.COM, Server not found in Kerberos database
I20260621 05:11:06.346570 12791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:11:06.341253 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.131.190:37467 (local address 127.3.131.130:47695)
0621 05:11:06.341434 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0621 05:11:06.341448 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0621 05:11:06.341545 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:11:06.341750 (+ 205us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:11:06.341751 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:11:06.341933 (+ 182us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:11:06.341977 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:11:06.341979 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:11:06.342695 (+ 716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:11:06.342697 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:11:06.343275 (+ 578us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:11:06.343277 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:11:06.343669 (+ 392us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:11:06.343670 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:11:06.343729 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:11:06.344309 (+ 580us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 05:11:06.344314 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:11:06.346467 (+ 2153us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.131.190:37467: Server oryx/127.3.131.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":155,"thread_start_us":85,"threads_started":1}
W20260621 05:11:06.346637 12790 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 05:11:06.341348 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.131.130:47695 (local address 127.3.131.190:37467)
0621 05:11:06.341440 (+ 92us) server_negotiation.cc:206] Beginning negotiation
0621 05:11:06.341442 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:11:06.341523 (+ 81us) server_negotiation.cc:407] Connection header received
0621 05:11:06.341647 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:11:06.341648 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:11:06.341662 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:11:06.341741 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:11:06.342065 (+ 324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:11:06.342604 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:11:06.343368 (+ 764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:11:06.343572 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:11:06.343669 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:11:06.346575 (+ 2906us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.131.130:47695: BlockingRecv error: recv got EOF from 127.3.131.130:47695 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
Jun 21 05:11:06 dist-test-slave-vrq1 krb5kdc[11752](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.131.131@KRBTEST.COM for oryx/127.3.131.190@KRBTEST.COM, Server not found in Kerberos database
Jun 21 05:11:06 dist-test-slave-vrq1 krb5kdc[11752](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.131.131@KRBTEST.COM for oryx/127.3.131.190@KRBTEST.COM, Server not found in Kerberos database
I20260621 05:11:06.499408 12792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:11:06.494049 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.131.190:37467 (local address 127.3.131.131:32943)
0621 05:11:06.494320 (+ 271us) negotiation.cc:107] Waiting for socket to connect
0621 05:11:06.494332 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0621 05:11:06.494404 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:11:06.494574 (+ 170us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:11:06.494576 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:11:06.494743 (+ 167us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:11:06.494826 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:11:06.494828 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:11:06.495585 (+ 757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:11:06.495586 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:11:06.496346 (+ 760us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:11:06.496349 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:11:06.496834 (+ 485us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:11:06.496835 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:11:06.496884 (+ 49us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:11:06.497453 (+ 569us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 05:11:06.497457 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:11:06.499345 (+ 1888us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.131.190:37467: Server oryx/127.3.131.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":249,"thread_start_us":82,"threads_started":1}
W20260621 05:11:06.499630 12790 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 05:11:06.494244 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.131.131:32943 (local address 127.3.131.190:37467)
0621 05:11:06.494323 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0621 05:11:06.494325 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:11:06.494403 (+ 78us) server_negotiation.cc:407] Connection header received
0621 05:11:06.494463 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:11:06.494463 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:11:06.494477 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:11:06.494523 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:11:06.494937 (+ 414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:11:06.495489 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:11:06.496451 (+ 962us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:11:06.496761 (+ 310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:11:06.496819 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:11:06.499562 (+ 2743us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.131.131:32943: BlockingRecv error: recv got EOF from 127.3.131.131:32943 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
Jun 21 05:11:07 dist-test-slave-vrq1 krb5kdc[11752](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.131.129@KRBTEST.COM for oryx/127.3.131.190@KRBTEST.COM, Server not found in Kerberos database
Jun 21 05:11:07 dist-test-slave-vrq1 krb5kdc[11752](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.131.129@KRBTEST.COM for oryx/127.3.131.190@KRBTEST.COM, Server not found in Kerberos database
I20260621 05:11:07.212904 12793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:11:07.207744 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.131.190:37467 (local address 127.3.131.129:50851)
0621 05:11:07.207945 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0621 05:11:07.207963 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0621 05:11:07.208029 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:11:07.208216 (+ 187us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:11:07.208217 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:11:07.208401 (+ 184us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:11:07.208451 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:11:07.208452 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:11:07.209241 (+ 789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:11:07.209242 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:11:07.209847 (+ 605us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:11:07.209849 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:11:07.210276 (+ 427us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:11:07.210276 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:11:07.210323 (+ 47us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:11:07.210866 (+ 543us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 05:11:07.210871 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:11:07.212831 (+ 1960us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.131.190:37467: Server oryx/127.3.131.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":178,"thread_start_us":74,"threads_started":1}
W20260621 05:11:07.213129 12794 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 05:11:07.207806 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.131.129:50851 (local address 127.3.131.190:37467)
0621 05:11:07.207977 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0621 05:11:07.207980 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0621 05:11:07.208016 (+ 36us) server_negotiation.cc:407] Connection header received
0621 05:11:07.208111 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:11:07.208111 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:11:07.208122 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:11:07.208177 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:11:07.208553 (+ 376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:11:07.209132 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:11:07.209953 (+ 821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:11:07.210195 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:11:07.210255 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:11:07.213058 (+ 2803us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.131.129:50851: BlockingRecv error: recv got EOF from 127.3.131.129:50851 (error 108)
Metrics: {"server-negotiator.queue_time_us":146,"thread_start_us":89,"threads_started":1}
Jun 21 05:11:07 dist-test-slave-vrq1 krb5kdc[11752](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.131.130@KRBTEST.COM for oryx/127.3.131.190@KRBTEST.COM, Server not found in Kerberos database
Jun 21 05:11:07 dist-test-slave-vrq1 krb5kdc[11752](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.131.130@KRBTEST.COM for oryx/127.3.131.190@KRBTEST.COM, Server not found in Kerberos database
I20260621 05:11:07.351960 12795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:11:07.347168 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.131.190:37467 (local address 127.3.131.130:57789)
0621 05:11:07.347320 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0621 05:11:07.347331 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0621 05:11:07.347399 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:11:07.347538 (+ 139us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:11:07.347538 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:11:07.347702 (+ 164us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:11:07.347748 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:11:07.347750 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:11:07.348412 (+ 662us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:11:07.348413 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:11:07.349083 (+ 670us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:11:07.349086 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:11:07.349464 (+ 378us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:11:07.349465 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:11:07.349521 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:11:07.350025 (+ 504us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 05:11:07.350030 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:11:07.351856 (+ 1826us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.131.190:37467: Server oryx/127.3.131.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":84,"threads_started":1}
W20260621 05:11:07.352065 12794 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 05:11:07.347239 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.131.130:57789 (local address 127.3.131.190:37467)
0621 05:11:07.347326 (+ 87us) server_negotiation.cc:206] Beginning negotiation
0621 05:11:07.347330 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 05:11:07.347383 (+ 53us) server_negotiation.cc:407] Connection header received
0621 05:11:07.347463 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:11:07.347463 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:11:07.347475 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:11:07.347518 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:11:07.347817 (+ 299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:11:07.348317 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:11:07.349186 (+ 869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:11:07.349372 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:11:07.349428 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:11:07.351966 (+ 2538us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.131.130:57789: BlockingRecv error: recv got EOF from 127.3.131.130:57789 (error 108)
Metrics: {"server-negotiator.queue_time_us":27}
Jun 21 05:11:07 dist-test-slave-vrq1 krb5kdc[11752](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.131.131@KRBTEST.COM for oryx/127.3.131.190@KRBTEST.COM, Server not found in Kerberos database
Jun 21 05:11:07 dist-test-slave-vrq1 krb5kdc[11752](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.131.131@KRBTEST.COM for oryx/127.3.131.190@KRBTEST.COM, Server not found in Kerberos database
I20260621 05:11:07.505445 12796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:11:07.499950 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.131.190:37467 (local address 127.3.131.131:39603)
0621 05:11:07.500138 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0621 05:11:07.500151 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0621 05:11:07.500445 (+ 294us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:11:07.500607 (+ 162us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:11:07.500608 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:11:07.500782 (+ 174us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:11:07.500832 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:11:07.500833 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:11:07.501668 (+ 835us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:11:07.501669 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:11:07.502294 (+ 625us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:11:07.502297 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:11:07.502725 (+ 428us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:11:07.502726 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:11:07.502781 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:11:07.503463 (+ 682us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 05:11:07.503468 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:11:07.505371 (+ 1903us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.131.190:37467: Server oryx/127.3.131.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":160,"thread_start_us":94,"threads_started":1}
W20260621 05:11:07.505626 12794 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 05:11:07.500017 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.131.131:39603 (local address 127.3.131.190:37467)
0621 05:11:07.500103 (+ 86us) server_negotiation.cc:206] Beginning negotiation
0621 05:11:07.500109 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 05:11:07.500449 (+ 340us) server_negotiation.cc:407] Connection header received
0621 05:11:07.500506 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:11:07.500506 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:11:07.500522 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:11:07.500574 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:11:07.500922 (+ 348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:11:07.501566 (+ 644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:11:07.502401 (+ 835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:11:07.502602 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:11:07.502669 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:11:07.505549 (+ 2880us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.131.131:39603: BlockingRecv error: recv got EOF from 127.3.131.131:39603 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
Jun 21 05:11:08 dist-test-slave-vrq1 krb5kdc[11752](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.131.129@KRBTEST.COM for oryx/127.3.131.190@KRBTEST.COM, Server not found in Kerberos database
Jun 21 05:11:08 dist-test-slave-vrq1 krb5kdc[11752](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.131.129@KRBTEST.COM for oryx/127.3.131.190@KRBTEST.COM, Server not found in Kerberos database
I20260621 05:11:08.218408 12797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:11:08.213475 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.131.190:37467 (local address 127.3.131.129:52251)
0621 05:11:08.213635 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0621 05:11:08.213646 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0621 05:11:08.213703 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:11:08.213839 (+ 136us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:11:08.213840 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:11:08.214010 (+ 170us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:11:08.214058 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:11:08.214059 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:11:08.214731 (+ 672us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:11:08.214732 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:11:08.215299 (+ 567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:11:08.215301 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:11:08.215694 (+ 393us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:11:08.215695 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:11:08.215732 (+ 37us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:11:08.216262 (+ 530us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 05:11:08.216267 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:11:08.218309 (+ 2042us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.131.190:37467: Server oryx/127.3.131.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":95,"threads_started":1}
W20260621 05:11:08.218472 12798 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 05:11:08.213575 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.131.129:52251 (local address 127.3.131.190:37467)
0621 05:11:08.213683 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0621 05:11:08.213685 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:11:08.213693 (+ 8us) server_negotiation.cc:407] Connection header received
0621 05:11:08.213752 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:11:08.213752 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:11:08.213762 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:11:08.213820 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:11:08.214125 (+ 305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:11:08.214628 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:11:08.215381 (+ 753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:11:08.215611 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:11:08.215695 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:11:08.218396 (+ 2701us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.131.129:52251: BlockingRecv error: recv got EOF from 127.3.131.129:52251 (error 108)
Metrics: {"server-negotiator.queue_time_us":83,"thread_start_us":49,"threads_started":1}
Jun 21 05:11:08 dist-test-slave-vrq1 krb5kdc[11752](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.131.130@KRBTEST.COM for oryx/127.3.131.190@KRBTEST.COM, Server not found in Kerberos database
Jun 21 05:11:08 dist-test-slave-vrq1 krb5kdc[11752](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.131.130@KRBTEST.COM for oryx/127.3.131.190@KRBTEST.COM, Server not found in Kerberos database
I20260621 05:11:08.357445 12799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:11:08.352599 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.131.190:37467 (local address 127.3.131.130:39299)
0621 05:11:08.352769 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0621 05:11:08.352792 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0621 05:11:08.352846 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:11:08.352954 (+ 108us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:11:08.352955 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:11:08.353131 (+ 176us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:11:08.353178 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:11:08.353179 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:11:08.353846 (+ 667us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:11:08.353847 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:11:08.354392 (+ 545us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:11:08.354393 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:11:08.354760 (+ 367us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:11:08.354761 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:11:08.354802 (+ 41us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:11:08.355328 (+ 526us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 05:11:08.355333 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:11:08.357372 (+ 2039us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.131.190:37467: Server oryx/127.3.131.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":82,"threads_started":1}
W20260621 05:11:08.357558 12798 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 05:11:08.352688 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.131.130:39299 (local address 127.3.131.190:37467)
0621 05:11:08.352744 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0621 05:11:08.352746 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:11:08.352837 (+ 91us) server_negotiation.cc:407] Connection header received
0621 05:11:08.352892 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:11:08.352893 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:11:08.352903 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:11:08.352951 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:11:08.353256 (+ 305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:11:08.353763 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:11:08.354488 (+ 725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:11:08.354687 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:11:08.354747 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:11:08.357484 (+ 2737us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.131.130:39299: BlockingRecv error: recv got EOF from 127.3.131.130:39299 (error 108)
Metrics: {"server-negotiator.queue_time_us":26}
Jun 21 05:11:08 dist-test-slave-vrq1 krb5kdc[11752](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.131.131@KRBTEST.COM for oryx/127.3.131.190@KRBTEST.COM, Server not found in Kerberos database
Jun 21 05:11:08 dist-test-slave-vrq1 krb5kdc[11752](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.131.131@KRBTEST.COM for oryx/127.3.131.190@KRBTEST.COM, Server not found in Kerberos database
I20260621 05:11:08.512009 12800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:11:08.506058 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.131.190:37467 (local address 127.3.131.131:59313)
0621 05:11:08.506218 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0621 05:11:08.506241 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0621 05:11:08.506316 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:11:08.506584 (+ 268us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:11:08.506593 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:11:08.506785 (+ 192us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:11:08.506835 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:11:08.506837 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:11:08.507675 (+ 838us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:11:08.507677 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:11:08.508299 (+ 622us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:11:08.508302 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:11:08.508722 (+ 420us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:11:08.508723 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:11:08.508799 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:11:08.509401 (+ 602us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 05:11:08.509407 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:11:08.511925 (+ 2518us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.131.190:37467: Server oryx/127.3.131.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":101,"threads_started":1}
W20260621 05:11:08.512200 12798 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 05:11:08.506162 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.131.131:59313 (local address 127.3.131.190:37467)
0621 05:11:08.506237 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0621 05:11:08.506240 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0621 05:11:08.506300 (+ 60us) server_negotiation.cc:407] Connection header received
0621 05:11:08.506368 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:11:08.506369 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:11:08.506384 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:11:08.506471 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:11:08.507015 (+ 544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:11:08.507566 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:11:08.508408 (+ 842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:11:08.508612 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:11:08.508679 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:11:08.512114 (+ 3435us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.131.131:59313: BlockingRecv error: recv got EOF from 127.3.131.131:59313 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
Jun 21 05:11:09 dist-test-slave-vrq1 krb5kdc[11752](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.131.129@KRBTEST.COM for oryx/127.3.131.190@KRBTEST.COM, Server not found in Kerberos database
Jun 21 05:11:09 dist-test-slave-vrq1 krb5kdc[11752](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.131.129@KRBTEST.COM for oryx/127.3.131.190@KRBTEST.COM, Server not found in Kerberos database
I20260621 05:11:09.225111 12801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:11:09.219019 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.131.190:37467 (local address 127.3.131.129:51873)
0621 05:11:09.219243 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0621 05:11:09.219255 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0621 05:11:09.219314 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:11:09.219679 (+ 365us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:11:09.219681 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:11:09.219858 (+ 177us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:11:09.219905 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:11:09.219907 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:11:09.220732 (+ 825us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:11:09.220733 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:11:09.221321 (+ 588us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:11:09.221323 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:11:09.221763 (+ 440us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:11:09.221764 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:11:09.221816 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:11:09.222567 (+ 751us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 05:11:09.222583 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:11:09.225040 (+ 2457us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.131.190:37467: Server oryx/127.3.131.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":200,"thread_start_us":90,"threads_started":1}
W20260621 05:11:09.225291 12802 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 05:11:09.219155 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.131.129:51873 (local address 127.3.131.190:37467)
0621 05:11:09.219377 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0621 05:11:09.219379 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:11:09.219389 (+ 10us) server_negotiation.cc:407] Connection header received
0621 05:11:09.219492 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:11:09.219493 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:11:09.219552 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:11:09.219629 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:11:09.220056 (+ 427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:11:09.220598 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:11:09.221455 (+ 857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:11:09.221668 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:11:09.221739 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:11:09.225183 (+ 3444us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.131.129:51873: BlockingRecv error: recv got EOF from 127.3.131.129:51873 (error 108)
Metrics: {"server-negotiator.queue_time_us":177,"thread_start_us":91,"threads_started":1}
Jun 21 05:11:09 dist-test-slave-vrq1 krb5kdc[11752](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.131.130@KRBTEST.COM for oryx/127.3.131.190@KRBTEST.COM, Server not found in Kerberos database
Jun 21 05:11:09 dist-test-slave-vrq1 krb5kdc[11752](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.131.130@KRBTEST.COM for oryx/127.3.131.190@KRBTEST.COM, Server not found in Kerberos database
I20260621 05:11:09.363168 12803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:11:09.358025 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.131.190:37467 (local address 127.3.131.130:36199)
0621 05:11:09.358230 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0621 05:11:09.358245 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0621 05:11:09.358309 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:11:09.358458 (+ 149us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:11:09.358459 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:11:09.358647 (+ 188us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:11:09.358695 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:11:09.358697 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:11:09.359447 (+ 750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:11:09.359449 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:11:09.359981 (+ 532us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:11:09.359983 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:11:09.360321 (+ 338us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:11:09.360322 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:11:09.360364 (+ 42us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:11:09.360860 (+ 496us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 05:11:09.360865 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:11:09.363104 (+ 2239us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.131.190:37467: Server oryx/127.3.131.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":171,"thread_start_us":139,"threads_started":1}
W20260621 05:11:09.363336 12802 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 05:11:09.358195 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.131.130:36199 (local address 127.3.131.190:37467)
0621 05:11:09.358245 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0621 05:11:09.358247 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:11:09.358294 (+ 47us) server_negotiation.cc:407] Connection header received
0621 05:11:09.358380 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:11:09.358380 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:11:09.358391 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:11:09.358469 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:11:09.358773 (+ 304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:11:09.359369 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:11:09.360064 (+ 695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:11:09.360253 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:11:09.360309 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:11:09.363256 (+ 2947us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.131.130:36199: BlockingRecv error: recv got EOF from 127.3.131.130:36199 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
Jun 21 05:11:09 dist-test-slave-vrq1 krb5kdc[11752](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.131.131@KRBTEST.COM for oryx/127.3.131.190@KRBTEST.COM, Server not found in Kerberos database
Jun 21 05:11:09 dist-test-slave-vrq1 krb5kdc[11752](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.131.131@KRBTEST.COM for oryx/127.3.131.190@KRBTEST.COM, Server not found in Kerberos database
I20260621 05:11:09.518122 12804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:11:09.512780 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.131.190:37467 (local address 127.3.131.131:60321)
0621 05:11:09.512968 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0621 05:11:09.512985 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0621 05:11:09.513056 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:11:09.513287 (+ 231us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:11:09.513289 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:11:09.513482 (+ 193us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:11:09.513533 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:11:09.513535 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:11:09.514300 (+ 765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:11:09.514301 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:11:09.514857 (+ 556us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:11:09.514859 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:11:09.515286 (+ 427us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:11:09.515287 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:11:09.515341 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:11:09.515970 (+ 629us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 05:11:09.515975 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:11:09.518064 (+ 2089us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.131.190:37467: Server oryx/127.3.131.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":163,"thread_start_us":81,"threads_started":1}
W20260621 05:11:09.518298 12802 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 05:11:09.512951 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.131.131:60321 (local address 127.3.131.190:37467)
0621 05:11:09.513044 (+ 93us) server_negotiation.cc:206] Beginning negotiation
0621 05:11:09.513046 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:11:09.513057 (+ 11us) server_negotiation.cc:407] Connection header received
0621 05:11:09.513136 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:11:09.513137 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:11:09.513151 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:11:09.513208 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:11:09.513640 (+ 432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:11:09.514203 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:11:09.514945 (+ 742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:11:09.515207 (+ 262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:11:09.515271 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:11:09.518179 (+ 2908us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.131.131:60321: BlockingRecv error: recv got EOF from 127.3.131.131:60321 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
Jun 21 05:11:10 dist-test-slave-vrq1 krb5kdc[11752](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.131.129@KRBTEST.COM for oryx/127.3.131.190@KRBTEST.COM, Server not found in Kerberos database
Jun 21 05:11:10 dist-test-slave-vrq1 krb5kdc[11752](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.131.129@KRBTEST.COM for oryx/127.3.131.190@KRBTEST.COM, Server not found in Kerberos database
I20260621 05:11:10.237676 12805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:11:10.225687 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.131.190:37467 (local address 127.3.131.129:46097)
0621 05:11:10.225814 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0621 05:11:10.225826 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0621 05:11:10.225885 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:11:10.226095 (+ 210us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:11:10.226096 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:11:10.226296 (+ 200us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:11:10.226350 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:11:10.226352 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:11:10.227433 (+ 1081us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:11:10.227434 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:11:10.228018 (+ 584us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:11:10.228020 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:11:10.228646 (+ 626us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:11:10.228647 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:11:10.228715 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:11:10.235325 (+ 6610us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 05:11:10.235331 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:11:10.237602 (+ 2271us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.131.190:37467: Server oryx/127.3.131.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":72,"threads_started":1}
W20260621 05:11:10.237905 12806 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 05:11:10.225753 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.131.129:46097 (local address 127.3.131.190:37467)
0621 05:11:10.225877 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0621 05:11:10.225880 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0621 05:11:10.225889 (+ 9us) server_negotiation.cc:407] Connection header received
0621 05:11:10.225958 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:11:10.225959 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:11:10.225972 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:11:10.226060 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:11:10.226434 (+ 374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:11:10.227329 (+ 895us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:11:10.228110 (+ 781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:11:10.228473 (+ 363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:11:10.228553 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:11:10.237760 (+ 9207us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.131.129:46097: BlockingRecv error: recv got EOF from 127.3.131.129:46097 (error 108)
Metrics: {"server-negotiator.queue_time_us":102,"thread_start_us":71,"threads_started":1}
Jun 21 05:11:10 dist-test-slave-vrq1 krb5kdc[11752](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.131.130@KRBTEST.COM for oryx/127.3.131.190@KRBTEST.COM, Server not found in Kerberos database
Jun 21 05:11:10 dist-test-slave-vrq1 krb5kdc[11752](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.131.130@KRBTEST.COM for oryx/127.3.131.190@KRBTEST.COM, Server not found in Kerberos database
I20260621 05:11:10.369220 12807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:11:10.363824 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.131.190:37467 (local address 127.3.131.130:46603)
0621 05:11:10.364026 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0621 05:11:10.364040 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0621 05:11:10.364126 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:11:10.364293 (+ 167us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:11:10.364294 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:11:10.364479 (+ 185us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:11:10.364546 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:11:10.364547 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:11:10.365386 (+ 839us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:11:10.365388 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:11:10.365953 (+ 565us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:11:10.365955 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:11:10.366424 (+ 469us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:11:10.366425 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:11:10.366534 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:11:10.367015 (+ 481us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 05:11:10.367020 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:11:10.369137 (+ 2117us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.131.190:37467: Server oryx/127.3.131.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":140,"threads_started":1}
W20260621 05:11:10.369385 12806 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 05:11:10.363962 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.131.130:46603 (local address 127.3.131.190:37467)
0621 05:11:10.364036 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0621 05:11:10.364038 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:11:10.364126 (+ 88us) server_negotiation.cc:407] Connection header received
0621 05:11:10.364198 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:11:10.364199 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:11:10.364211 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:11:10.364267 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:11:10.364665 (+ 398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:11:10.365259 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:11:10.366103 (+ 844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:11:10.366330 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:11:10.366393 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:11:10.369311 (+ 2918us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.131.130:46603: BlockingRecv error: recv got EOF from 127.3.131.130:46603 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
Jun 21 05:11:10 dist-test-slave-vrq1 krb5kdc[11752](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.131.131@KRBTEST.COM for oryx/127.3.131.190@KRBTEST.COM, Server not found in Kerberos database
Jun 21 05:11:10 dist-test-slave-vrq1 krb5kdc[11752](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.131.131@KRBTEST.COM for oryx/127.3.131.190@KRBTEST.COM, Server not found in Kerberos database
I20260621 05:11:10.524340 12808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:11:10.518713 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.131.190:37467 (local address 127.3.131.131:51489)
0621 05:11:10.518910 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0621 05:11:10.518921 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0621 05:11:10.518990 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:11:10.519220 (+ 230us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:11:10.519221 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:11:10.519394 (+ 173us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:11:10.519443 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:11:10.519445 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:11:10.520188 (+ 743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:11:10.520189 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:11:10.520756 (+ 567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:11:10.520758 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:11:10.521133 (+ 375us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:11:10.521134 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:11:10.521179 (+ 45us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:11:10.521880 (+ 701us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 05:11:10.521886 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:11:10.524228 (+ 2342us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.131.190:37467: Server oryx/127.3.131.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":116,"threads_started":1}
W20260621 05:11:10.524387 12806 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 05:11:10.518814 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.131.131:51489 (local address 127.3.131.190:37467)
0621 05:11:10.518914 (+ 100us) server_negotiation.cc:206] Beginning negotiation
0621 05:11:10.518916 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:11:10.518990 (+ 74us) server_negotiation.cc:407] Connection header received
0621 05:11:10.519042 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:11:10.519043 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:11:10.519067 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:11:10.519149 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:11:10.519548 (+ 399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:11:10.520092 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:11:10.520853 (+ 761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:11:10.521058 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:11:10.521121 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:11:10.524318 (+ 3197us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.131.131:51489: BlockingRecv error: recv got EOF from 127.3.131.131:51489 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
Jun 21 05:11:11 dist-test-slave-vrq1 krb5kdc[11752](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.131.129@KRBTEST.COM for oryx/127.3.131.190@KRBTEST.COM, Server not found in Kerberos database
Jun 21 05:11:11 dist-test-slave-vrq1 krb5kdc[11752](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.131.129@KRBTEST.COM for oryx/127.3.131.190@KRBTEST.COM, Server not found in Kerberos database
I20260621 05:11:11.244293 12809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:11:11.238382 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.131.190:37467 (local address 127.3.131.129:48311)
0621 05:11:11.238590 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0621 05:11:11.238605 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0621 05:11:11.238679 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:11:11.238921 (+ 242us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:11:11.238923 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:11:11.239113 (+ 190us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:11:11.239165 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:11:11.239167 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:11:11.240324 (+ 1157us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:11:11.240325 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:11:11.241043 (+ 718us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:11:11.241046 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:11:11.241507 (+ 461us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:11:11.241508 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:11:11.241566 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:11:11.242156 (+ 590us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 05:11:11.242161 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:11:11.244232 (+ 2071us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.131.190:37467: Server oryx/127.3.131.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":173,"threads_started":1}
W20260621 05:11:11.244601 12810 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 05:11:11.238598 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.131.129:48311 (local address 127.3.131.190:37467)
0621 05:11:11.238727 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0621 05:11:11.238729 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:11:11.238739 (+ 10us) server_negotiation.cc:407] Connection header received
0621 05:11:11.238778 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:11:11.238778 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:11:11.238789 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:11:11.238857 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:11:11.239396 (+ 539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:11:11.240166 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:11:11.241202 (+ 1036us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:11:11.241408 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:11:11.241471 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:11:11.244362 (+ 2891us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.131.129:48311: BlockingRecv error: recv got EOF from 127.3.131.129:48311 (error 108)
Metrics: {"server-negotiator.queue_time_us":105,"thread_start_us":74,"threads_started":1}
Jun 21 05:11:11 dist-test-slave-vrq1 krb5kdc[11752](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.131.130@KRBTEST.COM for oryx/127.3.131.190@KRBTEST.COM, Server not found in Kerberos database
Jun 21 05:11:11 dist-test-slave-vrq1 krb5kdc[11752](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.131.130@KRBTEST.COM for oryx/127.3.131.190@KRBTEST.COM, Server not found in Kerberos database
I20260621 05:11:11.375118 12811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:11:11.369895 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.131.190:37467 (local address 127.3.131.130:42307)
0621 05:11:11.370124 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0621 05:11:11.370158 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0621 05:11:11.370228 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:11:11.370400 (+ 172us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:11:11.370401 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:11:11.370567 (+ 166us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:11:11.370615 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:11:11.370617 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:11:11.371390 (+ 773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:11:11.371391 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:11:11.372007 (+ 616us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:11:11.372010 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:11:11.372550 (+ 540us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:11:11.372551 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:11:11.372619 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:11:11.373090 (+ 471us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 05:11:11.373095 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:11:11.374998 (+ 1903us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.131.190:37467: Server oryx/127.3.131.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":201,"thread_start_us":121,"threads_started":1}
W20260621 05:11:11.375149 12810 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 05:11:11.370027 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.131.130:42307 (local address 127.3.131.190:37467)
0621 05:11:11.370082 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0621 05:11:11.370084 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:11:11.370278 (+ 194us) server_negotiation.cc:407] Connection header received
0621 05:11:11.370316 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:11:11.370317 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:11:11.370329 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:11:11.370376 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:11:11.370718 (+ 342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:11:11.371287 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:11:11.372124 (+ 837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:11:11.372452 (+ 328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:11:11.372527 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:11:11.375084 (+ 2557us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.131.130:42307: BlockingRecv error: recv got EOF from 127.3.131.130:42307 (error 108)
Metrics: {"server-negotiator.queue_time_us":28}
I20260621 05:11:11.437397 3598 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWZOx2_/build/release/bin/kudu with pid 12335
I20260621 05:11:11.442075 3598 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWZOx2_/build/release/bin/kudu with pid 12487
I20260621 05:11:11.446542 3598 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWZOx2_/build/release/bin/kudu with pid 12619
I20260621 05:11:11.451045 3598 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWZOx2_/build/release/bin/kudu with pid 12265
2026-06-21T05:11:11Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (17602 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-06-21T05:11:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-21T05:11:11Z Disabled control of system clock
I20260621 05:11:11.479266 3598 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWZOx2_/build/release/bin/kudu
/tmp/dist-test-taskWZOx2_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782018616648578-3598-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782018616648578-3598-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782018616648578-3598-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782018616648578-3598-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.131.190:36809
--webserver_interface=127.3.131.190
--webserver_port=0
--builtin_ntp_servers=127.3.131.148:35181
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.131.190:36809
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260621 05:11:11.561265 12815 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 05:11:11.561497 12815 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 05:11:11.561527 12815 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 05:11:11.562745 12815 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 05:11:11.562788 12815 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 05:11:11.562800 12815 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 05:11:11.562813 12815 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 05:11:11.562826 12815 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 05:11:11.564296 12815 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.131.148:35181
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782018616648578-3598-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782018616648578-3598-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.131.190:36809
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.131.190:36809
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782018616648578-3598-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.131.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12815
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782018616648578-3598-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 21 Jun 2026 04:16:16 UTC on ccycloud.quasar-xycaha.root.comops.site
build id 4457881
I20260621 05:11:11.564519 12815 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 05:11:11.564776 12815 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 05:11:11.569773 12823 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:11:11.569901 12821 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:11:11.569952 12815 server_base.cc:1034] running on GCE node
W20260621 05:11:11.569890 12820 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:11:11.570382 12815 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 05:11:11.570685 12815 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 05:11:11.571882 12815 hybrid_clock.cc:648] HybridClock initialized: now 1782018671571871 us; error 43 us; skew 500 ppm
I20260621 05:11:11.573316 12815 webserver.cc:460] Webserver started at http://127.3.131.190:41329/ using document root <none> and password file <none>
I20260621 05:11:11.573599 12815 fs_manager.cc:359] Metadata directory not provided
I20260621 05:11:11.573699 12815 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 05:11:11.573859 12815 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 05:11:11.575074 12815 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782018616648578-3598-0/minicluster-data/master-0/data/instance:
uuid: "700485304416495f9525cba134eb430c"
format_stamp: "Formatted at 2026-06-21 05:11:11 on dist-test-slave-vrq1"
server_key: "ac33f16d1e2e83df465648396d3b3171"
server_key_iv: "97b907f23b45ef517d45b7ccd2c4f195"
server_key_version: "encryptionkey@0"
I20260621 05:11:11.575517 12815 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782018616648578-3598-0/minicluster-data/master-0/wal/instance:
uuid: "700485304416495f9525cba134eb430c"
format_stamp: "Formatted at 2026-06-21 05:11:11 on dist-test-slave-vrq1"
server_key: "ac33f16d1e2e83df465648396d3b3171"
server_key_iv: "97b907f23b45ef517d45b7ccd2c4f195"
server_key_version: "encryptionkey@0"
I20260621 05:11:11.576920 12815 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260621 05:11:11.577775 12829 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 05:11:11.578109 12815 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260621 05:11:11.578233 12815 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782018616648578-3598-0/minicluster-data/master-0/data,/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782018616648578-3598-0/minicluster-data/master-0/wal
uuid: "700485304416495f9525cba134eb430c"
format_stamp: "Formatted at 2026-06-21 05:11:11 on dist-test-slave-vrq1"
server_key: "ac33f16d1e2e83df465648396d3b3171"
server_key_iv: "97b907f23b45ef517d45b7ccd2c4f195"
server_key_version: "encryptionkey@0"
I20260621 05:11:11.578325 12815 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782018616648578-3598-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782018616648578-3598-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782018616648578-3598-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 05:11:11.591566 12815 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 05:11:11.591918 12815 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 05:11:11.592080 12815 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 05:11:11.596187 12815 rpc_server.cc:307] RPC server started. Bound to: 127.3.131.190:36809
I20260621 05:11:11.596232 12881 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.131.190:36809 every 8 connection(s)
I20260621 05:11:11.596621 12815 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782018616648578-3598-0/minicluster-data/master-0/data/info.pb
I20260621 05:11:11.597421 12882 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 05:11:11.600205 12882 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 700485304416495f9525cba134eb430c: Bootstrap starting.
I20260621 05:11:11.600803 12882 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 700485304416495f9525cba134eb430c: Neither blocks nor log segments found. Creating new log.
I20260621 05:11:11.601120 12882 log.cc:824] T 00000000000000000000000000000000 P 700485304416495f9525cba134eb430c: Log is configured to *not* fsync() on all Append() calls
I20260621 05:11:11.601802 12882 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 700485304416495f9525cba134eb430c: No bootstrap required, opened a new log
I20260621 05:11:11.603559 12882 raft_consensus.cc:348] T 00000000000000000000000000000000 P 700485304416495f9525cba134eb430c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "700485304416495f9525cba134eb430c" member_type: VOTER last_known_addr { host: "127.3.131.190" port: 36809 } }
I20260621 05:11:11.603701 12882 raft_consensus.cc:374] T 00000000000000000000000000000000 P 700485304416495f9525cba134eb430c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 05:11:11.603724 12882 raft_consensus.cc:729] T 00000000000000000000000000000000 P 700485304416495f9525cba134eb430c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 700485304416495f9525cba134eb430c, State: Initialized, Role: FOLLOWER
I20260621 05:11:11.603890 12882 consensus_queue.cc:260] T 00000000000000000000000000000000 P 700485304416495f9525cba134eb430c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "700485304416495f9525cba134eb430c" member_type: VOTER last_known_addr { host: "127.3.131.190" port: 36809 } }
I20260621 05:11:11.603971 12882 raft_consensus.cc:388] T 00000000000000000000000000000000 P 700485304416495f9525cba134eb430c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260621 05:11:11.604012 12882 raft_consensus.cc:482] T 00000000000000000000000000000000 P 700485304416495f9525cba134eb430c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260621 05:11:11.604059 12882 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 700485304416495f9525cba134eb430c [term 0 FOLLOWER]: Advancing to term 1
I20260621 05:11:11.604198 3598 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWZOx2_/build/release/bin/kudu as pid 12815
I20260621 05:11:11.604292 3598 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782018616648578-3598-0/minicluster-data/master-0/wal/instance
I20260621 05:11:11.604482 3598 external_mini_cluster.cc:1428] Setting key 8619db473404a9f56c7c621347111b5b
I20260621 05:11:11.604825 12882 raft_consensus.cc:504] T 00000000000000000000000000000000 P 700485304416495f9525cba134eb430c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "700485304416495f9525cba134eb430c" member_type: VOTER last_known_addr { host: "127.3.131.190" port: 36809 } }
I20260621 05:11:11.604938 12882 leader_election.cc:302] T 00000000000000000000000000000000 P 700485304416495f9525cba134eb430c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 700485304416495f9525cba134eb430c; no voters:
I20260621 05:11:11.605170 12882 leader_election.cc:288] T 00000000000000000000000000000000 P 700485304416495f9525cba134eb430c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260621 05:11:11.605250 12886 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 700485304416495f9525cba134eb430c [term 1 FOLLOWER]: Leader election won for term 1
I20260621 05:11:11.605398 12886 raft_consensus.cc:686] T 00000000000000000000000000000000 P 700485304416495f9525cba134eb430c [term 1 LEADER]: Becoming Leader. State: Replica: 700485304416495f9525cba134eb430c, State: Running, Role: LEADER
I20260621 05:11:11.605516 12882 sys_catalog.cc:564] T 00000000000000000000000000000000 P 700485304416495f9525cba134eb430c [sys.catalog]: configured and running, proceeding with master startup.
I20260621 05:11:11.605533 12886 consensus_queue.cc:237] T 00000000000000000000000000000000 P 700485304416495f9525cba134eb430c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "700485304416495f9525cba134eb430c" member_type: VOTER last_known_addr { host: "127.3.131.190" port: 36809 } }
I20260621 05:11:11.606128 12888 sys_catalog.cc:455] T 00000000000000000000000000000000 P 700485304416495f9525cba134eb430c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "700485304416495f9525cba134eb430c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "700485304416495f9525cba134eb430c" member_type: VOTER last_known_addr { host: "127.3.131.190" port: 36809 } } }
I20260621 05:11:11.606221 12888 sys_catalog.cc:458] T 00000000000000000000000000000000 P 700485304416495f9525cba134eb430c [sys.catalog]: This master's current role is: LEADER
I20260621 05:11:11.606418 12896 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260621 05:11:11.606460 12889 sys_catalog.cc:455] T 00000000000000000000000000000000 P 700485304416495f9525cba134eb430c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 700485304416495f9525cba134eb430c. Latest consensus state: current_term: 1 leader_uuid: "700485304416495f9525cba134eb430c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "700485304416495f9525cba134eb430c" member_type: VOTER last_known_addr { host: "127.3.131.190" port: 36809 } } }
I20260621 05:11:11.606551 12889 sys_catalog.cc:458] T 00000000000000000000000000000000 P 700485304416495f9525cba134eb430c [sys.catalog]: This master's current role is: LEADER
I20260621 05:11:11.607218 12896 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260621 05:11:11.608956 12896 catalog_manager.cc:1347] Generated new cluster ID: 42e74904599446f5a79cc149a8645b50
I20260621 05:11:11.609011 12896 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260621 05:11:11.611091 12887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:11:11.605008 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46058 (local address 127.3.131.190:36809)
0621 05:11:11.605454 (+ 446us) server_negotiation.cc:206] Beginning negotiation
0621 05:11:11.605462 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0621 05:11:11.605498 (+ 36us) server_negotiation.cc:407] Connection header received
0621 05:11:11.605648 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:11:11.605655 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:11:11.605734 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:11:11.605877 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:11:11.606122 (+ 245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:11:11.607686 (+ 1564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:11:11.608816 (+ 1130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:11:11.609321 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:11:11.609485 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:11:11.610310 (+ 825us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:11:11.610315 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:11:11.610315 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 05:11:11.610507 (+ 192us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:11:11.610542 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 05:11:11.610554 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:11:11.610643 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0621 05:11:11.610723 (+ 80us) server_negotiation.cc:299] Negotiation successful
0621 05:11:11.610928 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":204,"thread_start_us":128,"threads_started":1}
I20260621 05:11:11.630831 12896 catalog_manager.cc:1370] Generated new certificate authority record
I20260621 05:11:11.631477 12896 catalog_manager.cc:1504] Loading token signing keys...
I20260621 05:11:11.637866 12896 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 700485304416495f9525cba134eb430c: Generated new TSK 0
I20260621 05:11:11.638067 12896 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260621 05:11:11.644397 3598 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWZOx2_/build/release/bin/kudu
/tmp/dist-test-taskWZOx2_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782018616648578-3598-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782018616648578-3598-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782018616648578-3598-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782018616648578-3598-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.131.129:0
--local_ip_for_outbound_sockets=127.3.131.129
--webserver_interface=127.3.131.129
--webserver_port=0
--tserver_master_addrs=127.3.131.190:36809
--builtin_ntp_servers=127.3.131.148:35181
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260621 05:11:11.731663 12906 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 05:11:11.731931 12906 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 05:11:11.731969 12906 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 05:11:11.733224 12906 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 05:11:11.733294 12906 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 05:11:11.733341 12906 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.131.129
I20260621 05:11:11.734786 12906 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.131.148:35181
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782018616648578-3598-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782018616648578-3598-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.131.129:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782018616648578-3598-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.131.129
--webserver_port=0
--tserver_master_addrs=127.3.131.190:36809
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12906
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.131.129
--log_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782018616648578-3598-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 21 Jun 2026 04:16:16 UTC on ccycloud.quasar-xycaha.root.comops.site
build id 4457881
I20260621 05:11:11.735061 12906 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 05:11:11.735448 12906 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 05:11:11.740846 12911 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:11:11.740931 12914 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:11:11.740962 12906 server_base.cc:1034] running on GCE node
W20260621 05:11:11.741226 12912 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:11:11.741473 12906 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 05:11:11.741778 12906 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 05:11:11.742998 12906 hybrid_clock.cc:648] HybridClock initialized: now 1782018671742982 us; error 30 us; skew 500 ppm
I20260621 05:11:11.745038 12906 webserver.cc:460] Webserver started at http://127.3.131.129:40093/ using document root <none> and password file <none>
I20260621 05:11:11.745327 12906 fs_manager.cc:359] Metadata directory not provided
I20260621 05:11:11.745407 12906 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 05:11:11.745564 12906 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 05:11:11.746706 12906 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782018616648578-3598-0/minicluster-data/ts-0/data/instance:
uuid: "80df8a23b67f4e53b8d8f670a3bde88d"
format_stamp: "Formatted at 2026-06-21 05:11:11 on dist-test-slave-vrq1"
server_key: "c24b850ea16a2b0d1e28d2353b8b56d1"
server_key_iv: "9c6d1cb30f95e2c7d1c8482e50188cc9"
server_key_version: "encryptionkey@0"
I20260621 05:11:11.747095 12906 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782018616648578-3598-0/minicluster-data/ts-0/wal/instance:
uuid: "80df8a23b67f4e53b8d8f670a3bde88d"
format_stamp: "Formatted at 2026-06-21 05:11:11 on dist-test-slave-vrq1"
server_key: "c24b850ea16a2b0d1e28d2353b8b56d1"
server_key_iv: "9c6d1cb30f95e2c7d1c8482e50188cc9"
server_key_version: "encryptionkey@0"
I20260621 05:11:11.748523 12906 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260621 05:11:11.749402 12920 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 05:11:11.749730 12906 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260621 05:11:11.749871 12906 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782018616648578-3598-0/minicluster-data/ts-0/data,/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782018616648578-3598-0/minicluster-data/ts-0/wal
uuid: "80df8a23b67f4e53b8d8f670a3bde88d"
format_stamp: "Formatted at 2026-06-21 05:11:11 on dist-test-slave-vrq1"
server_key: "c24b850ea16a2b0d1e28d2353b8b56d1"
server_key_iv: "9c6d1cb30f95e2c7d1c8482e50188cc9"
server_key_version: "encryptionkey@0"
I20260621 05:11:11.749964 12906 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782018616648578-3598-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782018616648578-3598-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782018616648578-3598-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 05:11:11.776180 12906 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 05:11:11.776558 12906 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 05:11:11.776738 12906 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 05:11:11.777027 12906 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 05:11:11.777405 12906 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 05:11:11.777465 12906 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 05:11:11.777522 12906 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 05:11:11.777554 12906 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 05:11:11.784271 12906 rpc_server.cc:307] RPC server started. Bound to: 127.3.131.129:38097
I20260621 05:11:11.784365 13033 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.131.129:38097 every 8 connection(s)
I20260621 05:11:11.784739 12906 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782018616648578-3598-0/minicluster-data/ts-0/data/info.pb
I20260621 05:11:11.790500 3598 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWZOx2_/build/release/bin/kudu as pid 12906
I20260621 05:11:11.790621 3598 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782018616648578-3598-0/minicluster-data/ts-0/wal/instance
I20260621 05:11:11.790649 12887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:11:11.785767 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.131.129:36351 (local address 127.3.131.190:36809)
0621 05:11:11.785834 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0621 05:11:11.785836 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:11:11.786058 (+ 222us) server_negotiation.cc:407] Connection header received
0621 05:11:11.786627 (+ 569us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:11:11.786628 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:11:11.786644 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:11:11.786694 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:11:11.787438 (+ 744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:11:11.787959 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:11:11.789120 (+ 1161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:11:11.789343 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:11:11.789429 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:11:11.790349 (+ 920us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:11:11.790350 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:11:11.790350 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 05:11:11.790356 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:11:11.790370 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 05:11:11.790375 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:11:11.790428 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0621 05:11:11.790528 (+ 100us) server_negotiation.cc:299] Negotiation successful
0621 05:11:11.790564 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260621 05:11:11.790833 3598 external_mini_cluster.cc:1428] Setting key e861af248b4001273402f81f11a17cfb
I20260621 05:11:11.790897 13036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:11:11.785737 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.131.190:36809 (local address 127.3.131.129:36351)
0621 05:11:11.785929 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0621 05:11:11.785966 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0621 05:11:11.786508 (+ 542us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:11:11.786778 (+ 270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:11:11.786781 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:11:11.786790 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:11:11.787332 (+ 542us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:11:11.787336 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:11:11.788060 (+ 724us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:11:11.788062 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:11:11.789025 (+ 963us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:11:11.789027 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:11:11.789440 (+ 413us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:11:11.789441 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:11:11.789580 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:11:11.790185 (+ 605us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0621 05:11:11.790185 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:11:11.790216 (+ 31us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0621 05:11:11.790241 (+ 25us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 05:11:11.790434 (+ 193us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 05:11:11.790439 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 05:11:11.790443 (+ 4us) client_negotiation.cc:769] Sending connection context
0621 05:11:11.790532 (+ 89us) client_negotiation.cc:240] Negotiation successful
0621 05:11:11.790647 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":80,"threads_started":1}
I20260621 05:11:11.791436 13034 heartbeater.cc:347] Connected to a master server at 127.3.131.190:36809
I20260621 05:11:11.791579 13034 heartbeater.cc:464] Registering TS with master...
I20260621 05:11:11.791783 13034 heartbeater.cc:511] Master 127.3.131.190:36809 requested a full tablet report, sending...
I20260621 05:11:11.791841 3598 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWZOx2_/build/release/bin/kudu
/tmp/dist-test-taskWZOx2_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782018616648578-3598-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782018616648578-3598-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782018616648578-3598-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782018616648578-3598-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.131.130:0
--local_ip_for_outbound_sockets=127.3.131.130
--webserver_interface=127.3.131.130
--webserver_port=0
--tserver_master_addrs=127.3.131.190:36809
--builtin_ntp_servers=127.3.131.148:35181
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260621 05:11:11.792325 12846 ts_manager.cc:194] Registered new tserver with Master: 80df8a23b67f4e53b8d8f670a3bde88d (127.3.131.129:38097)
I20260621 05:11:11.793224 12846 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.3.131.129:36351
W20260621 05:11:11.876452 13037 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 05:11:11.876690 13037 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 05:11:11.876715 13037 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 05:11:11.877897 13037 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 05:11:11.877939 13037 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 05:11:11.877965 13037 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.131.130
I20260621 05:11:11.879432 13037 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.131.148:35181
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782018616648578-3598-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782018616648578-3598-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.131.130:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782018616648578-3598-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.131.130
--webserver_port=0
--tserver_master_addrs=127.3.131.190:36809
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13037
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.131.130
--log_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782018616648578-3598-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 21 Jun 2026 04:16:16 UTC on ccycloud.quasar-xycaha.root.comops.site
build id 4457881
I20260621 05:11:11.879657 13037 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 05:11:11.879922 13037 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260621 05:11:11.885769 13037 server_base.cc:1034] running on GCE node
W20260621 05:11:11.885741 13045 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:11:11.885710 13043 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:11:11.885713 13042 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:11:11.886262 13037 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 05:11:11.886495 13037 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 05:11:11.887693 13037 hybrid_clock.cc:648] HybridClock initialized: now 1782018671887669 us; error 44 us; skew 500 ppm
I20260621 05:11:11.889113 13037 webserver.cc:460] Webserver started at http://127.3.131.130:37773/ using document root <none> and password file <none>
I20260621 05:11:11.889329 13037 fs_manager.cc:359] Metadata directory not provided
I20260621 05:11:11.889372 13037 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 05:11:11.889472 13037 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 05:11:11.890524 13037 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782018616648578-3598-0/minicluster-data/ts-1/data/instance:
uuid: "c156446dad324472a36de536dd575faa"
format_stamp: "Formatted at 2026-06-21 05:11:11 on dist-test-slave-vrq1"
server_key: "2e789409cd6558dc934acb3a462597ce"
server_key_iv: "b2ed44116418c23f5594cc8b8ed0a6df"
server_key_version: "encryptionkey@0"
I20260621 05:11:11.890823 13037 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782018616648578-3598-0/minicluster-data/ts-1/wal/instance:
uuid: "c156446dad324472a36de536dd575faa"
format_stamp: "Formatted at 2026-06-21 05:11:11 on dist-test-slave-vrq1"
server_key: "2e789409cd6558dc934acb3a462597ce"
server_key_iv: "b2ed44116418c23f5594cc8b8ed0a6df"
server_key_version: "encryptionkey@0"
I20260621 05:11:11.892242 13037 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260621 05:11:11.893105 13051 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 05:11:11.893343 13037 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260621 05:11:11.893452 13037 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782018616648578-3598-0/minicluster-data/ts-1/data,/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782018616648578-3598-0/minicluster-data/ts-1/wal
uuid: "c156446dad324472a36de536dd575faa"
format_stamp: "Formatted at 2026-06-21 05:11:11 on dist-test-slave-vrq1"
server_key: "2e789409cd6558dc934acb3a462597ce"
server_key_iv: "b2ed44116418c23f5594cc8b8ed0a6df"
server_key_version: "encryptionkey@0"
I20260621 05:11:11.893574 13037 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782018616648578-3598-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782018616648578-3598-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782018616648578-3598-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 05:11:11.911115 13037 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 05:11:11.911531 13037 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 05:11:11.911693 13037 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 05:11:11.911942 13037 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 05:11:11.912292 13037 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 05:11:11.912348 13037 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 05:11:11.912393 13037 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 05:11:11.912418 13037 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 05:11:11.918831 13037 rpc_server.cc:307] RPC server started. Bound to: 127.3.131.130:42009
I20260621 05:11:11.918874 13164 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.131.130:42009 every 8 connection(s)
I20260621 05:11:11.919267 13037 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782018616648578-3598-0/minicluster-data/ts-1/data/info.pb
I20260621 05:11:11.925235 12887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:11:11.920071 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.131.130:58699 (local address 127.3.131.190:36809)
0621 05:11:11.920142 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0621 05:11:11.920145 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0621 05:11:11.920503 (+ 358us) server_negotiation.cc:407] Connection header received
0621 05:11:11.921092 (+ 589us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:11:11.921094 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:11:11.921105 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:11:11.921165 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:11:11.921897 (+ 732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:11:11.922421 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:11:11.923613 (+ 1192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:11:11.923809 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:11:11.923878 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:11:11.924852 (+ 974us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:11:11.924854 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:11:11.924855 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 05:11:11.924860 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:11:11.924875 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 05:11:11.924880 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:11:11.924936 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0621 05:11:11.925141 (+ 205us) server_negotiation.cc:299] Negotiation successful
0621 05:11:11.925170 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260621 05:11:11.925374 13167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:11:11.920070 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.131.190:36809 (local address 127.3.131.130:58699)
0621 05:11:11.920357 (+ 287us) negotiation.cc:107] Waiting for socket to connect
0621 05:11:11.920394 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0621 05:11:11.920918 (+ 524us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:11:11.921267 (+ 349us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:11:11.921271 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:11:11.921280 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:11:11.921804 (+ 524us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:11:11.921809 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:11:11.922546 (+ 737us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:11:11.922547 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:11:11.923503 (+ 956us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:11:11.923506 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:11:11.923940 (+ 434us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:11:11.923941 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:11:11.924122 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:11:11.924679 (+ 557us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0621 05:11:11.924679 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:11:11.924709 (+ 30us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0621 05:11:11.924734 (+ 25us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 05:11:11.924970 (+ 236us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 05:11:11.924976 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 05:11:11.924979 (+ 3us) client_negotiation.cc:769] Sending connection context
0621 05:11:11.925103 (+ 124us) client_negotiation.cc:240] Negotiation successful
0621 05:11:11.925193 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":221,"thread_start_us":94,"threads_started":1}
I20260621 05:11:11.925959 13165 heartbeater.cc:347] Connected to a master server at 127.3.131.190:36809
I20260621 05:11:11.926082 13165 heartbeater.cc:464] Registering TS with master...
I20260621 05:11:11.926321 13165 heartbeater.cc:511] Master 127.3.131.190:36809 requested a full tablet report, sending...
I20260621 05:11:11.926740 12846 ts_manager.cc:194] Registered new tserver with Master: c156446dad324472a36de536dd575faa (127.3.131.130:42009)
I20260621 05:11:11.927239 12846 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.3.131.130:58699
I20260621 05:11:11.927882 3598 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWZOx2_/build/release/bin/kudu as pid 13037
I20260621 05:11:11.927976 3598 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782018616648578-3598-0/minicluster-data/ts-1/wal/instance
I20260621 05:11:11.928166 3598 external_mini_cluster.cc:1428] Setting key 0452be23e74f72f6b960e1106c0fbde4
I20260621 05:11:11.929207 3598 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWZOx2_/build/release/bin/kudu
/tmp/dist-test-taskWZOx2_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782018616648578-3598-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782018616648578-3598-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782018616648578-3598-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782018616648578-3598-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.131.131:0
--local_ip_for_outbound_sockets=127.3.131.131
--webserver_interface=127.3.131.131
--webserver_port=0
--tserver_master_addrs=127.3.131.190:36809
--builtin_ntp_servers=127.3.131.148:35181
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260621 05:11:12.016431 13168 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 05:11:12.016664 13168 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 05:11:12.016687 13168 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 05:11:12.017918 13168 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 05:11:12.017967 13168 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 05:11:12.017994 13168 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.131.131
I20260621 05:11:12.019464 13168 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.131.148:35181
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782018616648578-3598-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782018616648578-3598-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.131.131:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782018616648578-3598-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.131.131
--webserver_port=0
--tserver_master_addrs=127.3.131.190:36809
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13168
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.131.131
--log_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782018616648578-3598-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 21 Jun 2026 04:16:16 UTC on ccycloud.quasar-xycaha.root.comops.site
build id 4457881
I20260621 05:11:12.019733 13168 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 05:11:12.020056 13168 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 05:11:12.025959 13176 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:11:12.026324 13173 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:11:12.026343 13174 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:11:12.026610 13168 server_base.cc:1034] running on GCE node
I20260621 05:11:12.026881 13168 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 05:11:12.027226 13168 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 05:11:12.028434 13168 hybrid_clock.cc:648] HybridClock initialized: now 1782018672028407 us; error 44 us; skew 500 ppm
I20260621 05:11:12.030027 13168 webserver.cc:460] Webserver started at http://127.3.131.131:42141/ using document root <none> and password file <none>
I20260621 05:11:12.030315 13168 fs_manager.cc:359] Metadata directory not provided
I20260621 05:11:12.030390 13168 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 05:11:12.030548 13168 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 05:11:12.031773 13168 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782018616648578-3598-0/minicluster-data/ts-2/data/instance:
uuid: "00f8e2f06f56433cb6783d5a4e90a23a"
format_stamp: "Formatted at 2026-06-21 05:11:12 on dist-test-slave-vrq1"
server_key: "c7df099543b923d97996bfe2ba6a22e4"
server_key_iv: "f6c1163b68baf0b303c6645cd5d7f7f6"
server_key_version: "encryptionkey@0"
I20260621 05:11:12.032162 13168 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782018616648578-3598-0/minicluster-data/ts-2/wal/instance:
uuid: "00f8e2f06f56433cb6783d5a4e90a23a"
format_stamp: "Formatted at 2026-06-21 05:11:12 on dist-test-slave-vrq1"
server_key: "c7df099543b923d97996bfe2ba6a22e4"
server_key_iv: "f6c1163b68baf0b303c6645cd5d7f7f6"
server_key_version: "encryptionkey@0"
I20260621 05:11:12.033639 13168 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260621 05:11:12.034675 13182 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 05:11:12.035048 13168 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260621 05:11:12.035223 13168 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782018616648578-3598-0/minicluster-data/ts-2/data,/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782018616648578-3598-0/minicluster-data/ts-2/wal
uuid: "00f8e2f06f56433cb6783d5a4e90a23a"
format_stamp: "Formatted at 2026-06-21 05:11:12 on dist-test-slave-vrq1"
server_key: "c7df099543b923d97996bfe2ba6a22e4"
server_key_iv: "f6c1163b68baf0b303c6645cd5d7f7f6"
server_key_version: "encryptionkey@0"
I20260621 05:11:12.035346 13168 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782018616648578-3598-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782018616648578-3598-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782018616648578-3598-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 05:11:12.052950 13168 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 05:11:12.053427 13168 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 05:11:12.053629 13168 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 05:11:12.054052 13168 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 05:11:12.054469 13168 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 05:11:12.054670 13168 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 05:11:12.054736 13168 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 05:11:12.054764 13168 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 05:11:12.061517 13168 rpc_server.cc:307] RPC server started. Bound to: 127.3.131.131:41263
I20260621 05:11:12.061558 13295 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.131.131:41263 every 8 connection(s)
I20260621 05:11:12.062026 13168 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782018616648578-3598-0/minicluster-data/ts-2/data/info.pb
I20260621 05:11:12.064781 3598 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWZOx2_/build/release/bin/kudu as pid 13168
I20260621 05:11:12.064860 3598 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782018616648578-3598-0/minicluster-data/ts-2/wal/instance
I20260621 05:11:12.065054 3598 external_mini_cluster.cc:1428] Setting key edf523bf699309f353bc95c8904008ce
I20260621 05:11:12.068444 12887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:11:12.062869 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.131.131:43901 (local address 127.3.131.190:36809)
0621 05:11:12.062975 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0621 05:11:12.062978 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0621 05:11:12.063265 (+ 287us) server_negotiation.cc:407] Connection header received
0621 05:11:12.063785 (+ 520us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:11:12.063786 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:11:12.063797 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:11:12.063855 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:11:12.064639 (+ 784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:11:12.065383 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:11:12.066607 (+ 1224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:11:12.066819 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:11:12.066878 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:11:12.068070 (+ 1192us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:11:12.068072 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:11:12.068072 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 05:11:12.068077 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:11:12.068101 (+ 24us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 05:11:12.068106 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:11:12.068171 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0621 05:11:12.068341 (+ 170us) server_negotiation.cc:299] Negotiation successful
0621 05:11:12.068376 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260621 05:11:12.068638 13298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:11:12.062777 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.131.190:36809 (local address 127.3.131.131:43901)
0621 05:11:12.063009 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0621 05:11:12.063054 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0621 05:11:12.063664 (+ 610us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:11:12.063933 (+ 269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:11:12.063937 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:11:12.063946 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:11:12.064501 (+ 555us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:11:12.064505 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:11:12.065500 (+ 995us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:11:12.065501 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:11:12.066511 (+ 1010us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:11:12.066514 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:11:12.066919 (+ 405us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:11:12.066920 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:11:12.067047 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:11:12.067899 (+ 852us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0621 05:11:12.067899 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:11:12.067936 (+ 37us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0621 05:11:12.067969 (+ 33us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 05:11:12.068215 (+ 246us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 05:11:12.068222 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 05:11:12.068225 (+ 3us) client_negotiation.cc:769] Sending connection context
0621 05:11:12.068301 (+ 76us) client_negotiation.cc:240] Negotiation successful
0621 05:11:12.068397 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":82,"threads_started":1}
I20260621 05:11:12.069223 13296 heartbeater.cc:347] Connected to a master server at 127.3.131.190:36809
I20260621 05:11:12.069325 13296 heartbeater.cc:464] Registering TS with master...
I20260621 05:11:12.069485 13296 heartbeater.cc:511] Master 127.3.131.190:36809 requested a full tablet report, sending...
I20260621 05:11:12.069895 12846 ts_manager.cc:194] Registered new tserver with Master: 00f8e2f06f56433cb6783d5a4e90a23a (127.3.131.131:41263)
I20260621 05:11:12.070299 12846 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.3.131.131:43901
I20260621 05:11:12.077172 3598 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260621 05:11:12.079381 12887 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 05:11:12.078783 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46064 (local address 127.3.131.190:36809)
0621 05:11:12.078861 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0621 05:11:12.078863 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:11:12.078905 (+ 42us) server_negotiation.cc:407] Connection header received
0621 05:11:12.078968 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:11:12.078969 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:11:12.078983 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:11:12.079041 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:11:12.079277 (+ 236us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46064: BlockingRecv error: recv got EOF from 127.0.0.1:46064 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
I20260621 05:11:12.079828 3598 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWZOx2_/build/release/bin/kudu with pid 12906
I20260621 05:11:12.084810 3598 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWZOx2_/build/release/bin/kudu with pid 13037
I20260621 05:11:12.089699 3598 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWZOx2_/build/release/bin/kudu with pid 13168
I20260621 05:11:12.094301 3598 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWZOx2_/build/release/bin/kudu with pid 12815
2026-06-21T05:11:12Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (642 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-06-21T05:11:12Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-21T05:11:12Z Disabled control of system clock
I20260621 05:11:12.122215 3598 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWZOx2_/build/release/bin/kudu
/tmp/dist-test-taskWZOx2_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782018616648578-3598-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782018616648578-3598-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782018616648578-3598-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782018616648578-3598-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.131.190:40607
--webserver_interface=127.3.131.190
--webserver_port=0
--builtin_ntp_servers=127.3.131.148:44449
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.131.190:40607
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260621 05:11:12.205634 13307 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 05:11:12.205868 13307 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 05:11:12.205899 13307 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 05:11:12.207142 13307 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 05:11:12.207206 13307 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 05:11:12.207257 13307 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 05:11:12.207284 13307 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 05:11:12.207299 13307 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 05:11:12.208648 13307 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.131.148:44449
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782018616648578-3598-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782018616648578-3598-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.131.190:40607
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.131.190:40607
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782018616648578-3598-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.131.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13307
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782018616648578-3598-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 21 Jun 2026 04:16:16 UTC on ccycloud.quasar-xycaha.root.comops.site
build id 4457881
I20260621 05:11:12.208865 13307 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 05:11:12.209127 13307 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 05:11:12.214536 13313 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:11:12.214525 13315 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:11:12.214572 13312 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:11:12.215047 13307 server_base.cc:1034] running on GCE node
I20260621 05:11:12.215368 13307 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 05:11:12.215711 13307 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 05:11:12.216859 13307 hybrid_clock.cc:648] HybridClock initialized: now 1782018672216832 us; error 42 us; skew 500 ppm
I20260621 05:11:12.218364 13307 webserver.cc:460] Webserver started at http://127.3.131.190:42465/ using document root <none> and password file <none>
I20260621 05:11:12.218649 13307 fs_manager.cc:359] Metadata directory not provided
I20260621 05:11:12.218724 13307 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 05:11:12.218876 13307 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 05:11:12.220064 13307 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782018616648578-3598-0/minicluster-data/master-0/data/instance:
uuid: "d7ae95c707404f0ea88b376d321ab927"
format_stamp: "Formatted at 2026-06-21 05:11:12 on dist-test-slave-vrq1"
server_key: "70f85c4cfa9922cb661c86448925fd79"
server_key_iv: "d2a915ea7ad7b62bc8101ebcac241930"
server_key_version: "encryptionkey@0"
I20260621 05:11:12.220438 13307 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782018616648578-3598-0/minicluster-data/master-0/wal/instance:
uuid: "d7ae95c707404f0ea88b376d321ab927"
format_stamp: "Formatted at 2026-06-21 05:11:12 on dist-test-slave-vrq1"
server_key: "70f85c4cfa9922cb661c86448925fd79"
server_key_iv: "d2a915ea7ad7b62bc8101ebcac241930"
server_key_version: "encryptionkey@0"
I20260621 05:11:12.221840 13307 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260621 05:11:12.222782 13321 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 05:11:12.222987 13307 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260621 05:11:12.223094 13307 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782018616648578-3598-0/minicluster-data/master-0/data,/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782018616648578-3598-0/minicluster-data/master-0/wal
uuid: "d7ae95c707404f0ea88b376d321ab927"
format_stamp: "Formatted at 2026-06-21 05:11:12 on dist-test-slave-vrq1"
server_key: "70f85c4cfa9922cb661c86448925fd79"
server_key_iv: "d2a915ea7ad7b62bc8101ebcac241930"
server_key_version: "encryptionkey@0"
I20260621 05:11:12.223157 13307 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782018616648578-3598-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782018616648578-3598-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782018616648578-3598-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 05:11:12.226615 13307 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 05:11:12.226850 13307 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 05:11:12.227003 13307 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 05:11:12.230903 13307 rpc_server.cc:307] RPC server started. Bound to: 127.3.131.190:40607
I20260621 05:11:12.230935 13373 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.131.190:40607 every 8 connection(s)
I20260621 05:11:12.231343 13307 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782018616648578-3598-0/minicluster-data/master-0/data/info.pb
I20260621 05:11:12.232003 13374 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 05:11:12.234757 13374 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d7ae95c707404f0ea88b376d321ab927: Bootstrap starting.
I20260621 05:11:12.235426 13374 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d7ae95c707404f0ea88b376d321ab927: Neither blocks nor log segments found. Creating new log.
I20260621 05:11:12.235764 13374 log.cc:824] T 00000000000000000000000000000000 P d7ae95c707404f0ea88b376d321ab927: Log is configured to *not* fsync() on all Append() calls
I20260621 05:11:12.236475 13374 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d7ae95c707404f0ea88b376d321ab927: No bootstrap required, opened a new log
I20260621 05:11:12.237205 3598 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWZOx2_/build/release/bin/kudu as pid 13307
I20260621 05:11:12.237284 3598 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782018616648578-3598-0/minicluster-data/master-0/wal/instance
I20260621 05:11:12.237493 3598 external_mini_cluster.cc:1428] Setting key 5ad27666d0b308e14c36ac6ea30fd753
I20260621 05:11:12.238623 13374 raft_consensus.cc:348] T 00000000000000000000000000000000 P d7ae95c707404f0ea88b376d321ab927 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d7ae95c707404f0ea88b376d321ab927" member_type: VOTER last_known_addr { host: "127.3.131.190" port: 40607 } }
I20260621 05:11:12.238801 13374 raft_consensus.cc:374] T 00000000000000000000000000000000 P d7ae95c707404f0ea88b376d321ab927 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 05:11:12.238840 13374 raft_consensus.cc:729] T 00000000000000000000000000000000 P d7ae95c707404f0ea88b376d321ab927 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d7ae95c707404f0ea88b376d321ab927, State: Initialized, Role: FOLLOWER
I20260621 05:11:12.239001 13374 consensus_queue.cc:260] T 00000000000000000000000000000000 P d7ae95c707404f0ea88b376d321ab927 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d7ae95c707404f0ea88b376d321ab927" member_type: VOTER last_known_addr { host: "127.3.131.190" port: 40607 } }
I20260621 05:11:12.239118 13374 raft_consensus.cc:388] T 00000000000000000000000000000000 P d7ae95c707404f0ea88b376d321ab927 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260621 05:11:12.239151 13374 raft_consensus.cc:482] T 00000000000000000000000000000000 P d7ae95c707404f0ea88b376d321ab927 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260621 05:11:12.239267 13374 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d7ae95c707404f0ea88b376d321ab927 [term 0 FOLLOWER]: Advancing to term 1
I20260621 05:11:12.240006 13374 raft_consensus.cc:504] T 00000000000000000000000000000000 P d7ae95c707404f0ea88b376d321ab927 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d7ae95c707404f0ea88b376d321ab927" member_type: VOTER last_known_addr { host: "127.3.131.190" port: 40607 } }
I20260621 05:11:12.240099 13374 leader_election.cc:302] T 00000000000000000000000000000000 P d7ae95c707404f0ea88b376d321ab927 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d7ae95c707404f0ea88b376d321ab927; no voters:
I20260621 05:11:12.240149 13378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:11:12.238093 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50606 (local address 127.3.131.190:40607)
0621 05:11:12.238313 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0621 05:11:12.238321 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0621 05:11:12.238345 (+ 24us) server_negotiation.cc:407] Connection header received
0621 05:11:12.238602 (+ 257us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:11:12.238609 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:11:12.238647 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:11:12.238765 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:11:12.239590 (+ 825us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:11:12.239593 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:11:12.239594 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 05:11:12.239597 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:11:12.239648 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 05:11:12.239659 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:11:12.239724 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0621 05:11:12.239812 (+ 88us) server_negotiation.cc:299] Negotiation successful
0621 05:11:12.239887 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":148,"thread_start_us":104,"threads_started":1}
I20260621 05:11:12.240270 13374 leader_election.cc:288] T 00000000000000000000000000000000 P d7ae95c707404f0ea88b376d321ab927 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260621 05:11:12.240538 13374 sys_catalog.cc:564] T 00000000000000000000000000000000 P d7ae95c707404f0ea88b376d321ab927 [sys.catalog]: configured and running, proceeding with master startup.
I20260621 05:11:12.240595 13379 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d7ae95c707404f0ea88b376d321ab927 [term 1 FOLLOWER]: Leader election won for term 1
I20260621 05:11:12.240702 13379 raft_consensus.cc:686] T 00000000000000000000000000000000 P d7ae95c707404f0ea88b376d321ab927 [term 1 LEADER]: Becoming Leader. State: Replica: d7ae95c707404f0ea88b376d321ab927, State: Running, Role: LEADER
I20260621 05:11:12.240789 13379 consensus_queue.cc:237] T 00000000000000000000000000000000 P d7ae95c707404f0ea88b376d321ab927 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d7ae95c707404f0ea88b376d321ab927" member_type: VOTER last_known_addr { host: "127.3.131.190" port: 40607 } }
I20260621 05:11:12.241348 13381 sys_catalog.cc:455] T 00000000000000000000000000000000 P d7ae95c707404f0ea88b376d321ab927 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d7ae95c707404f0ea88b376d321ab927. Latest consensus state: current_term: 1 leader_uuid: "d7ae95c707404f0ea88b376d321ab927" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d7ae95c707404f0ea88b376d321ab927" member_type: VOTER last_known_addr { host: "127.3.131.190" port: 40607 } } }
I20260621 05:11:12.241345 13380 sys_catalog.cc:455] T 00000000000000000000000000000000 P d7ae95c707404f0ea88b376d321ab927 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d7ae95c707404f0ea88b376d321ab927" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d7ae95c707404f0ea88b376d321ab927" member_type: VOTER last_known_addr { host: "127.3.131.190" port: 40607 } } }
I20260621 05:11:12.241465 13381 sys_catalog.cc:458] T 00000000000000000000000000000000 P d7ae95c707404f0ea88b376d321ab927 [sys.catalog]: This master's current role is: LEADER
I20260621 05:11:12.241473 13380 sys_catalog.cc:458] T 00000000000000000000000000000000 P d7ae95c707404f0ea88b376d321ab927 [sys.catalog]: This master's current role is: LEADER
I20260621 05:11:12.241663 13388 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260621 05:11:12.242173 13388 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260621 05:11:12.243711 13388 catalog_manager.cc:1347] Generated new cluster ID: 2cab5c555b2b4a1685d98eff39c82163
I20260621 05:11:12.243780 13388 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260621 05:11:12.248760 13388 catalog_manager.cc:1370] Generated new certificate authority record
I20260621 05:11:12.248926 13388 catalog_manager.cc:1504] Loading token signing keys...
I20260621 05:11:12.252794 13388 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d7ae95c707404f0ea88b376d321ab927: Generated new TSK 0
I20260621 05:11:12.252992 13388 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260621 05:11:12.260845 3598 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWZOx2_/build/release/bin/kudu
/tmp/dist-test-taskWZOx2_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782018616648578-3598-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782018616648578-3598-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782018616648578-3598-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782018616648578-3598-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.131.129:0
--local_ip_for_outbound_sockets=127.3.131.129
--webserver_interface=127.3.131.129
--webserver_port=0
--tserver_master_addrs=127.3.131.190:40607
--builtin_ntp_servers=127.3.131.148:44449
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260621 05:11:12.346441 13398 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 05:11:12.346683 13398 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 05:11:12.346720 13398 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 05:11:12.347950 13398 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 05:11:12.348018 13398 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 05:11:12.348062 13398 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.131.129
I20260621 05:11:12.349460 13398 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.131.148:44449
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782018616648578-3598-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782018616648578-3598-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.131.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782018616648578-3598-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.131.129
--webserver_port=0
--tserver_master_addrs=127.3.131.190:40607
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13398
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.131.129
--log_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782018616648578-3598-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 21 Jun 2026 04:16:16 UTC on ccycloud.quasar-xycaha.root.comops.site
build id 4457881
I20260621 05:11:12.349741 13398 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 05:11:12.350041 13398 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 05:11:12.356161 13403 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:11:12.356202 13406 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:11:12.356201 13404 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:11:12.356460 13398 server_base.cc:1034] running on GCE node
I20260621 05:11:12.356805 13398 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 05:11:12.357107 13398 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 05:11:12.358286 13398 hybrid_clock.cc:648] HybridClock initialized: now 1782018672358259 us; error 38 us; skew 500 ppm
I20260621 05:11:12.359819 13398 webserver.cc:460] Webserver started at http://127.3.131.129:41749/ using document root <none> and password file <none>
I20260621 05:11:12.360088 13398 fs_manager.cc:359] Metadata directory not provided
I20260621 05:11:12.360145 13398 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 05:11:12.360277 13398 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 05:11:12.361685 13398 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782018616648578-3598-0/minicluster-data/ts-0/data/instance:
uuid: "05677cb75e3e4b9c81af8f1498523f9d"
format_stamp: "Formatted at 2026-06-21 05:11:12 on dist-test-slave-vrq1"
server_key: "fa8f7e51f5fa3a261643d50444354754"
server_key_iv: "e8026a2634e3b4999f6b1e1347b72547"
server_key_version: "encryptionkey@0"
I20260621 05:11:12.362093 13398 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782018616648578-3598-0/minicluster-data/ts-0/wal/instance:
uuid: "05677cb75e3e4b9c81af8f1498523f9d"
format_stamp: "Formatted at 2026-06-21 05:11:12 on dist-test-slave-vrq1"
server_key: "fa8f7e51f5fa3a261643d50444354754"
server_key_iv: "e8026a2634e3b4999f6b1e1347b72547"
server_key_version: "encryptionkey@0"
I20260621 05:11:12.363822 13398 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.004s sys 0.000s
I20260621 05:11:12.364715 13412 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 05:11:12.364939 13398 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260621 05:11:12.365032 13398 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782018616648578-3598-0/minicluster-data/ts-0/data,/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782018616648578-3598-0/minicluster-data/ts-0/wal
uuid: "05677cb75e3e4b9c81af8f1498523f9d"
format_stamp: "Formatted at 2026-06-21 05:11:12 on dist-test-slave-vrq1"
server_key: "fa8f7e51f5fa3a261643d50444354754"
server_key_iv: "e8026a2634e3b4999f6b1e1347b72547"
server_key_version: "encryptionkey@0"
I20260621 05:11:12.365113 13398 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782018616648578-3598-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782018616648578-3598-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782018616648578-3598-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 05:11:12.370366 13398 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 05:11:12.370716 13398 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 05:11:12.370868 13398 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 05:11:12.371107 13398 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 05:11:12.371542 13398 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 05:11:12.371587 13398 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 05:11:12.371616 13398 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 05:11:12.371634 13398 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 05:11:12.378472 13398 rpc_server.cc:307] RPC server started. Bound to: 127.3.131.129:37219
I20260621 05:11:12.378566 13525 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.131.129:37219 every 8 connection(s)
I20260621 05:11:12.378953 13398 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782018616648578-3598-0/minicluster-data/ts-0/data/info.pb
I20260621 05:11:12.382835 13378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:11:12.379962 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.131.129:35765 (local address 127.3.131.190:40607)
0621 05:11:12.380233 (+ 271us) server_negotiation.cc:206] Beginning negotiation
0621 05:11:12.380236 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0621 05:11:12.380452 (+ 216us) server_negotiation.cc:407] Connection header received
0621 05:11:12.381086 (+ 634us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:11:12.381088 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:11:12.381094 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:11:12.381156 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:11:12.382558 (+ 1402us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:11:12.382560 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:11:12.382560 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 05:11:12.382565 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:11:12.382584 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 05:11:12.382589 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:11:12.382654 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0621 05:11:12.382760 (+ 106us) server_negotiation.cc:299] Negotiation successful
0621 05:11:12.382775 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":241}
I20260621 05:11:12.383054 13528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:11:12.379961 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.131.190:40607 (local address 127.3.131.129:35765)
0621 05:11:12.380321 (+ 360us) negotiation.cc:107] Waiting for socket to connect
0621 05:11:12.380362 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0621 05:11:12.380950 (+ 588us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:11:12.381241 (+ 291us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:11:12.381246 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:11:12.381256 (+ 10us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:11:12.382349 (+ 1093us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0621 05:11:12.382349 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:11:12.382400 (+ 51us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0621 05:11:12.382437 (+ 37us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 05:11:12.382675 (+ 238us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 05:11:12.382681 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 05:11:12.382684 (+ 3us) client_negotiation.cc:769] Sending connection context
0621 05:11:12.382763 (+ 79us) client_negotiation.cc:240] Negotiation successful
0621 05:11:12.382831 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":299,"thread_start_us":95,"threads_started":1}
I20260621 05:11:12.383603 13526 heartbeater.cc:347] Connected to a master server at 127.3.131.190:40607
I20260621 05:11:12.383720 13526 heartbeater.cc:464] Registering TS with master...
I20260621 05:11:12.383875 13526 heartbeater.cc:511] Master 127.3.131.190:40607 requested a full tablet report, sending...
I20260621 05:11:12.384548 13338 ts_manager.cc:194] Registered new tserver with Master: 05677cb75e3e4b9c81af8f1498523f9d (127.3.131.129:37219)
I20260621 05:11:12.386301 3598 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWZOx2_/build/release/bin/kudu as pid 13398
I20260621 05:11:12.386377 3598 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782018616648578-3598-0/minicluster-data/ts-0/wal/instance
I20260621 05:11:12.386515 3598 external_mini_cluster.cc:1428] Setting key d0a5547bdfd0100c3c69ff2e6e1f6d7e
I20260621 05:11:12.387645 3598 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWZOx2_/build/release/bin/kudu
/tmp/dist-test-taskWZOx2_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782018616648578-3598-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782018616648578-3598-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782018616648578-3598-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782018616648578-3598-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.131.130:0
--local_ip_for_outbound_sockets=127.3.131.130
--webserver_interface=127.3.131.130
--webserver_port=0
--tserver_master_addrs=127.3.131.190:40607
--builtin_ntp_servers=127.3.131.148:44449
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260621 05:11:12.471437 13529 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 05:11:12.471678 13529 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 05:11:12.471704 13529 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 05:11:12.472975 13529 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 05:11:12.473021 13529 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 05:11:12.473052 13529 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.131.130
I20260621 05:11:12.474790 13529 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.131.148:44449
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782018616648578-3598-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782018616648578-3598-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.131.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782018616648578-3598-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.131.130
--webserver_port=0
--tserver_master_addrs=127.3.131.190:40607
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13529
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.131.130
--log_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782018616648578-3598-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 21 Jun 2026 04:16:16 UTC on ccycloud.quasar-xycaha.root.comops.site
build id 4457881
I20260621 05:11:12.475157 13529 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 05:11:12.475528 13529 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 05:11:12.481179 13534 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:11:12.481277 13535 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:11:12.481287 13537 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:11:12.481427 13529 server_base.cc:1034] running on GCE node
I20260621 05:11:12.481689 13529 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 05:11:12.481953 13529 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 05:11:12.483246 13529 hybrid_clock.cc:648] HybridClock initialized: now 1782018672483217 us; error 60 us; skew 500 ppm
I20260621 05:11:12.484754 13529 webserver.cc:460] Webserver started at http://127.3.131.130:35003/ using document root <none> and password file <none>
I20260621 05:11:12.485029 13529 fs_manager.cc:359] Metadata directory not provided
I20260621 05:11:12.485113 13529 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 05:11:12.485266 13529 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 05:11:12.486421 13529 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782018616648578-3598-0/minicluster-data/ts-1/data/instance:
uuid: "e58ae9c4433741bfa8d26ac15a115ff0"
format_stamp: "Formatted at 2026-06-21 05:11:12 on dist-test-slave-vrq1"
server_key: "46d1d6168a8d306bcc85c843a4e8476f"
server_key_iv: "e8f7f18649f8b465479d1a7c9fb150d4"
server_key_version: "encryptionkey@0"
I20260621 05:11:12.486799 13529 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782018616648578-3598-0/minicluster-data/ts-1/wal/instance:
uuid: "e58ae9c4433741bfa8d26ac15a115ff0"
format_stamp: "Formatted at 2026-06-21 05:11:12 on dist-test-slave-vrq1"
server_key: "46d1d6168a8d306bcc85c843a4e8476f"
server_key_iv: "e8f7f18649f8b465479d1a7c9fb150d4"
server_key_version: "encryptionkey@0"
I20260621 05:11:12.488370 13529 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260621 05:11:12.489305 13543 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 05:11:12.489529 13529 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260621 05:11:12.489641 13529 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782018616648578-3598-0/minicluster-data/ts-1/data,/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782018616648578-3598-0/minicluster-data/ts-1/wal
uuid: "e58ae9c4433741bfa8d26ac15a115ff0"
format_stamp: "Formatted at 2026-06-21 05:11:12 on dist-test-slave-vrq1"
server_key: "46d1d6168a8d306bcc85c843a4e8476f"
server_key_iv: "e8f7f18649f8b465479d1a7c9fb150d4"
server_key_version: "encryptionkey@0"
I20260621 05:11:12.489735 13529 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782018616648578-3598-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782018616648578-3598-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782018616648578-3598-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 05:11:12.493367 13529 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 05:11:12.493643 13529 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 05:11:12.493777 13529 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 05:11:12.493975 13529 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 05:11:12.494303 13529 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 05:11:12.494356 13529 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 05:11:12.494396 13529 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 05:11:12.494423 13529 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 05:11:12.500227 13529 rpc_server.cc:307] RPC server started. Bound to: 127.3.131.130:46327
I20260621 05:11:12.500279 13656 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.131.130:46327 every 8 connection(s)
I20260621 05:11:12.500646 13529 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782018616648578-3598-0/minicluster-data/ts-1/data/info.pb
I20260621 05:11:12.503428 3598 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWZOx2_/build/release/bin/kudu as pid 13529
I20260621 05:11:12.503589 3598 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782018616648578-3598-0/minicluster-data/ts-1/wal/instance
I20260621 05:11:12.503777 3598 external_mini_cluster.cc:1428] Setting key 6cfbfc3ca0a71a41e6afe2698ec26d45
I20260621 05:11:12.503924 13378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:11:12.501419 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.131.130:35963 (local address 127.3.131.190:40607)
0621 05:11:12.501480 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0621 05:11:12.501482 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:11:12.501782 (+ 300us) server_negotiation.cc:407] Connection header received
0621 05:11:12.502203 (+ 421us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:11:12.502204 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:11:12.502210 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:11:12.502277 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:11:12.503616 (+ 1339us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:11:12.503618 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:11:12.503618 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 05:11:12.503624 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:11:12.503643 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 05:11:12.503648 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:11:12.503732 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0621 05:11:12.503824 (+ 92us) server_negotiation.cc:299] Negotiation successful
0621 05:11:12.503842 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260621 05:11:12.504154 13659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:11:12.501389 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.131.190:40607 (local address 127.3.131.130:35963)
0621 05:11:12.501582 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0621 05:11:12.501623 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0621 05:11:12.502089 (+ 466us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:11:12.502384 (+ 295us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:11:12.502388 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:11:12.502398 (+ 10us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:11:12.503425 (+ 1027us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0621 05:11:12.503426 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:11:12.503457 (+ 31us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0621 05:11:12.503487 (+ 30us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 05:11:12.503727 (+ 240us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 05:11:12.503733 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 05:11:12.503737 (+ 4us) client_negotiation.cc:769] Sending connection context
0621 05:11:12.503838 (+ 101us) client_negotiation.cc:240] Negotiation successful
0621 05:11:12.503906 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":75,"threads_started":1}
I20260621 05:11:12.504573 13657 heartbeater.cc:347] Connected to a master server at 127.3.131.190:40607
I20260621 05:11:12.504686 13657 heartbeater.cc:464] Registering TS with master...
I20260621 05:11:12.504834 13657 heartbeater.cc:511] Master 127.3.131.190:40607 requested a full tablet report, sending...
I20260621 05:11:12.504830 3598 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWZOx2_/build/release/bin/kudu
/tmp/dist-test-taskWZOx2_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782018616648578-3598-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782018616648578-3598-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782018616648578-3598-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782018616648578-3598-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.131.131:0
--local_ip_for_outbound_sockets=127.3.131.131
--webserver_interface=127.3.131.131
--webserver_port=0
--tserver_master_addrs=127.3.131.190:40607
--builtin_ntp_servers=127.3.131.148:44449
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260621 05:11:12.505231 13338 ts_manager.cc:194] Registered new tserver with Master: e58ae9c4433741bfa8d26ac15a115ff0 (127.3.131.130:46327)
W20260621 05:11:12.592530 13660 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 05:11:12.592818 13660 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 05:11:12.592859 13660 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 05:11:12.594275 13660 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 05:11:12.594347 13660 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 05:11:12.594408 13660 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.131.131
I20260621 05:11:12.595927 13660 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.131.148:44449
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782018616648578-3598-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782018616648578-3598-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.131.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782018616648578-3598-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.131.131
--webserver_port=0
--tserver_master_addrs=127.3.131.190:40607
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13660
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.131.131
--log_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782018616648578-3598-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 21 Jun 2026 04:16:16 UTC on ccycloud.quasar-xycaha.root.comops.site
build id 4457881
I20260621 05:11:12.596223 13660 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 05:11:12.596546 13660 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 05:11:12.602088 13666 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:11:12.602089 13668 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:11:12.602088 13665 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:11:12.602443 13660 server_base.cc:1034] running on GCE node
I20260621 05:11:12.602730 13660 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 05:11:12.602996 13660 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 05:11:12.604189 13660 hybrid_clock.cc:648] HybridClock initialized: now 1782018672604171 us; error 34 us; skew 500 ppm
I20260621 05:11:12.605945 13660 webserver.cc:460] Webserver started at http://127.3.131.131:36147/ using document root <none> and password file <none>
I20260621 05:11:12.606241 13660 fs_manager.cc:359] Metadata directory not provided
I20260621 05:11:12.606315 13660 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 05:11:12.606477 13660 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 05:11:12.607707 13660 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782018616648578-3598-0/minicluster-data/ts-2/data/instance:
uuid: "f087eb0c79524a21b4424ea900446da3"
format_stamp: "Formatted at 2026-06-21 05:11:12 on dist-test-slave-vrq1"
server_key: "93b8955b9a6d676c6a7bfb6ade4326d4"
server_key_iv: "f7f5d6345b378339e329932a301aecaa"
server_key_version: "encryptionkey@0"
I20260621 05:11:12.608090 13660 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782018616648578-3598-0/minicluster-data/ts-2/wal/instance:
uuid: "f087eb0c79524a21b4424ea900446da3"
format_stamp: "Formatted at 2026-06-21 05:11:12 on dist-test-slave-vrq1"
server_key: "93b8955b9a6d676c6a7bfb6ade4326d4"
server_key_iv: "f7f5d6345b378339e329932a301aecaa"
server_key_version: "encryptionkey@0"
I20260621 05:11:12.609618 13660 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260621 05:11:12.610482 13674 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 05:11:12.610772 13660 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260621 05:11:12.610885 13660 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782018616648578-3598-0/minicluster-data/ts-2/data,/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782018616648578-3598-0/minicluster-data/ts-2/wal
uuid: "f087eb0c79524a21b4424ea900446da3"
format_stamp: "Formatted at 2026-06-21 05:11:12 on dist-test-slave-vrq1"
server_key: "93b8955b9a6d676c6a7bfb6ade4326d4"
server_key_iv: "f7f5d6345b378339e329932a301aecaa"
server_key_version: "encryptionkey@0"
I20260621 05:11:12.610975 13660 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782018616648578-3598-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782018616648578-3598-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782018616648578-3598-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 05:11:12.614543 13660 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 05:11:12.614851 13660 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 05:11:12.614995 13660 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 05:11:12.615265 13660 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 05:11:12.615620 13660 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 05:11:12.615695 13660 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 05:11:12.615737 13660 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 05:11:12.615763 13660 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 05:11:12.622566 13660 rpc_server.cc:307] RPC server started. Bound to: 127.3.131.131:39981
I20260621 05:11:12.622655 13787 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.131.131:39981 every 8 connection(s)
I20260621 05:11:12.623047 13660 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782018616648578-3598-0/minicluster-data/ts-2/data/info.pb
I20260621 05:11:12.626698 13378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:11:12.624404 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.131.131:48147 (local address 127.3.131.190:40607)
0621 05:11:12.624465 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0621 05:11:12.624467 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:11:12.624479 (+ 12us) server_negotiation.cc:407] Connection header received
0621 05:11:12.624993 (+ 514us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:11:12.624995 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:11:12.625001 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:11:12.625092 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:11:12.626344 (+ 1252us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:11:12.626346 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:11:12.626346 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 05:11:12.626351 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:11:12.626367 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 05:11:12.626372 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:11:12.626436 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0621 05:11:12.626554 (+ 118us) server_negotiation.cc:299] Negotiation successful
0621 05:11:12.626571 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260621 05:11:12.626801 13790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:11:12.623972 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.131.190:40607 (local address 127.3.131.131:48147)
0621 05:11:12.624211 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0621 05:11:12.624283 (+ 72us) client_negotiation.cc:174] Beginning negotiation
0621 05:11:12.624820 (+ 537us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:11:12.625150 (+ 330us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:11:12.625153 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:11:12.625162 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:11:12.626183 (+ 1021us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0621 05:11:12.626184 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:11:12.626214 (+ 30us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0621 05:11:12.626247 (+ 33us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 05:11:12.626435 (+ 188us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 05:11:12.626437 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 05:11:12.626440 (+ 3us) client_negotiation.cc:769] Sending connection context
0621 05:11:12.626496 (+ 56us) client_negotiation.cc:240] Negotiation successful
0621 05:11:12.626548 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":97,"threads_started":1}
I20260621 05:11:12.627432 13788 heartbeater.cc:347] Connected to a master server at 127.3.131.190:40607
I20260621 05:11:12.627552 13788 heartbeater.cc:464] Registering TS with master...
I20260621 05:11:12.627704 13788 heartbeater.cc:511] Master 127.3.131.190:40607 requested a full tablet report, sending...
I20260621 05:11:12.628242 13338 ts_manager.cc:194] Registered new tserver with Master: f087eb0c79524a21b4424ea900446da3 (127.3.131.131:39981)
I20260621 05:11:12.630278 3598 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWZOx2_/build/release/bin/kudu as pid 13660
I20260621 05:11:12.630383 3598 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782018616648578-3598-0/minicluster-data/ts-2/wal/instance
I20260621 05:11:12.630610 3598 external_mini_cluster.cc:1428] Setting key b992bf71b0474d464051d140f4690cfe
I20260621 05:11:12.631775 3598 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260621 05:11:12.633643 13378 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 05:11:12.633184 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50616 (local address 127.3.131.190:40607)
0621 05:11:12.633255 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0621 05:11:12.633257 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:11:12.633371 (+ 114us) server_negotiation.cc:407] Connection header received
0621 05:11:12.633394 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:11:12.633395 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:11:12.633399 (+ 4us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:11:12.633461 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:11:12.633560 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50616: BlockingRecv error: recv got EOF from 127.0.0.1:50616 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
I20260621 05:11:12.633724 3598 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWZOx2_/build/release/bin/kudu with pid 13398
I20260621 05:11:12.638787 3598 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWZOx2_/build/release/bin/kudu with pid 13529
I20260621 05:11:12.644299 3598 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWZOx2_/build/release/bin/kudu with pid 13660
I20260621 05:11:12.649248 3598 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWZOx2_/build/release/bin/kudu with pid 13307
2026-06-21T05:11:12Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (555 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782018616648578-3598-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 21 05:11:12 dist-test-slave-vrq1 krb5kdc[13798](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 21 05:11:12 dist-test-slave-vrq1 krb5kdc[13798](info): set up 2 sockets
Jun 21 05:11:12 dist-test-slave-vrq1 krb5kdc[13798](info): commencing operation
krb5kdc: starting...
W20260621 05:11:13.696534 3598 mini_kdc.cc:121] Time spent starting KDC: real 1.027s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782018616648578-3598-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782018616648578-3598-0/krb5kdc/test-user.keytab.
Jun 21 05:11:13 dist-test-slave-vrq1 krb5kdc[13798](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782018673, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-21T05:11:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-21T05:11:13Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.131.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.131.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.131.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782018616648578-3598-0/krb5kdc/kudu_127.3.131.190.keytab.
Entry for principal kudu/127.3.131.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782018616648578-3598-0/krb5kdc/kudu_127.3.131.190.keytab.
I20260621 05:11:13.827435 3598 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWZOx2_/build/release/bin/kudu
/tmp/dist-test-taskWZOx2_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782018616648578-3598-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782018616648578-3598-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782018616648578-3598-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782018616648578-3598-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.131.190:45897
--webserver_interface=127.3.131.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782018616648578-3598-0/krb5kdc/kudu_127.3.131.190.keytab
--principal=kudu/127.3.131.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.131.148:37439
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.131.190:45897
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782018616648578-3598-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782018616648578-3598-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782018616648578-3598-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 05:11:13.909334 13811 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 05:11:13.909611 13811 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 05:11:13.909653 13811 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 05:11:13.910866 13811 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 05:11:13.910938 13811 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 05:11:13.910964 13811 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 05:11:13.910984 13811 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 05:11:13.911015 13811 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 05:11:13.912581 13811 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.131.148:37439
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782018616648578-3598-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782018616648578-3598-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.131.190:45897
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782018616648578-3598-0/krb5kdc/kudu_127.3.131.190.keytab
--principal=kudu/127.3.131.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.131.190:45897
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782018616648578-3598-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.131.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13811
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782018616648578-3598-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 21 Jun 2026 04:16:16 UTC on ccycloud.quasar-xycaha.root.comops.site
build id 4457881
I20260621 05:11:13.912875 13811 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 05:11:13.913110 13811 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 05:11:13.918193 13819 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:11:13.918206 13817 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:11:13.918248 13811 server_base.cc:1034] running on GCE node
W20260621 05:11:13.918215 13816 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:11:13.918687 13811 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 05:11:13.918943 13811 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 05:11:13.920087 13811 hybrid_clock.cc:648] HybridClock initialized: now 1782018673920036 us; error 62 us; skew 500 ppm
Jun 21 05:11:13 dist-test-slave-vrq1 krb5kdc[13798](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782018673, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.131.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 05:11:13.922760 13811 init.cc:377] Logged in from keytab as kudu/127.3.131.190@KRBTEST.COM (short username kudu)
I20260621 05:11:13.923323 13811 webserver.cc:460] Webserver started at http://127.3.131.190:36047/ using document root <none> and password file <none>
I20260621 05:11:13.923554 13811 fs_manager.cc:359] Metadata directory not provided
I20260621 05:11:13.923602 13811 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 05:11:13.923700 13811 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 05:11:13.924767 13811 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782018616648578-3598-0/minicluster-data/master-0/data/instance:
uuid: "a234a64c229b4e818f0422e3740c8ff5"
format_stamp: "Formatted at 2026-06-21 05:11:13 on dist-test-slave-vrq1"
server_key: "d0ccaa53ff57ae18dc345827612fb0c5"
server_key_iv: "969145703a1cca8838ed6d4823f1e7e3"
server_key_version: "encryptionkey@0"
I20260621 05:11:13.925109 13811 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782018616648578-3598-0/minicluster-data/master-0/wal/instance:
uuid: "a234a64c229b4e818f0422e3740c8ff5"
format_stamp: "Formatted at 2026-06-21 05:11:13 on dist-test-slave-vrq1"
server_key: "d0ccaa53ff57ae18dc345827612fb0c5"
server_key_iv: "969145703a1cca8838ed6d4823f1e7e3"
server_key_version: "encryptionkey@0"
I20260621 05:11:13.926544 13811 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260621 05:11:13.927361 13826 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 05:11:13.927521 13811 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260621 05:11:13.927608 13811 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782018616648578-3598-0/minicluster-data/master-0/data,/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782018616648578-3598-0/minicluster-data/master-0/wal
uuid: "a234a64c229b4e818f0422e3740c8ff5"
format_stamp: "Formatted at 2026-06-21 05:11:13 on dist-test-slave-vrq1"
server_key: "d0ccaa53ff57ae18dc345827612fb0c5"
server_key_iv: "969145703a1cca8838ed6d4823f1e7e3"
server_key_version: "encryptionkey@0"
I20260621 05:11:13.927691 13811 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782018616648578-3598-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782018616648578-3598-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782018616648578-3598-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 05:11:13.944084 13811 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 05:11:13.946657 13811 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 05:11:13.946820 13811 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 05:11:13.950785 13811 rpc_server.cc:307] RPC server started. Bound to: 127.3.131.190:45897
I20260621 05:11:13.950822 13878 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.131.190:45897 every 8 connection(s)
I20260621 05:11:13.951270 13811 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782018616648578-3598-0/minicluster-data/master-0/data/info.pb
I20260621 05:11:13.952275 13879 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 05:11:13.952939 3598 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWZOx2_/build/release/bin/kudu as pid 13811
I20260621 05:11:13.953040 3598 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782018616648578-3598-0/minicluster-data/master-0/wal/instance
I20260621 05:11:13.953243 3598 external_mini_cluster.cc:1428] Setting key fae68079d57d8432f61e720d4b059aef
I20260621 05:11:13.955406 13879 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a234a64c229b4e818f0422e3740c8ff5: Bootstrap starting.
I20260621 05:11:13.956077 13879 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a234a64c229b4e818f0422e3740c8ff5: Neither blocks nor log segments found. Creating new log.
I20260621 05:11:13.956406 13879 log.cc:824] T 00000000000000000000000000000000 P a234a64c229b4e818f0422e3740c8ff5: Log is configured to *not* fsync() on all Append() calls
I20260621 05:11:13.957170 13879 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a234a64c229b4e818f0422e3740c8ff5: No bootstrap required, opened a new log
I20260621 05:11:13.959277 13879 raft_consensus.cc:348] T 00000000000000000000000000000000 P a234a64c229b4e818f0422e3740c8ff5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a234a64c229b4e818f0422e3740c8ff5" member_type: VOTER last_known_addr { host: "127.3.131.190" port: 45897 } }
I20260621 05:11:13.959492 13879 raft_consensus.cc:374] T 00000000000000000000000000000000 P a234a64c229b4e818f0422e3740c8ff5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 05:11:13.959532 13879 raft_consensus.cc:729] T 00000000000000000000000000000000 P a234a64c229b4e818f0422e3740c8ff5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a234a64c229b4e818f0422e3740c8ff5, State: Initialized, Role: FOLLOWER
I20260621 05:11:13.959689 13879 consensus_queue.cc:260] T 00000000000000000000000000000000 P a234a64c229b4e818f0422e3740c8ff5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a234a64c229b4e818f0422e3740c8ff5" member_type: VOTER last_known_addr { host: "127.3.131.190" port: 45897 } }
I20260621 05:11:13.959801 13879 raft_consensus.cc:388] T 00000000000000000000000000000000 P a234a64c229b4e818f0422e3740c8ff5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260621 05:11:13.959838 13879 raft_consensus.cc:482] T 00000000000000000000000000000000 P a234a64c229b4e818f0422e3740c8ff5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260621 05:11:13.959890 13879 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a234a64c229b4e818f0422e3740c8ff5 [term 0 FOLLOWER]: Advancing to term 1
Jun 21 05:11:13 dist-test-slave-vrq1 krb5kdc[13798](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782018673, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.131.190@KRBTEST.COM
I20260621 05:11:13.960668 13879 raft_consensus.cc:504] T 00000000000000000000000000000000 P a234a64c229b4e818f0422e3740c8ff5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a234a64c229b4e818f0422e3740c8ff5" member_type: VOTER last_known_addr { host: "127.3.131.190" port: 45897 } }
I20260621 05:11:13.960796 13879 leader_election.cc:302] T 00000000000000000000000000000000 P a234a64c229b4e818f0422e3740c8ff5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a234a64c229b4e818f0422e3740c8ff5; no voters:
I20260621 05:11:13.961007 13879 leader_election.cc:288] T 00000000000000000000000000000000 P a234a64c229b4e818f0422e3740c8ff5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260621 05:11:13.961094 13884 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a234a64c229b4e818f0422e3740c8ff5 [term 1 FOLLOWER]: Leader election won for term 1
I20260621 05:11:13.961313 13879 sys_catalog.cc:564] T 00000000000000000000000000000000 P a234a64c229b4e818f0422e3740c8ff5 [sys.catalog]: configured and running, proceeding with master startup.
I20260621 05:11:13.961609 13884 raft_consensus.cc:686] T 00000000000000000000000000000000 P a234a64c229b4e818f0422e3740c8ff5 [term 1 LEADER]: Becoming Leader. State: Replica: a234a64c229b4e818f0422e3740c8ff5, State: Running, Role: LEADER
I20260621 05:11:13.961728 13884 consensus_queue.cc:237] T 00000000000000000000000000000000 P a234a64c229b4e818f0422e3740c8ff5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a234a64c229b4e818f0422e3740c8ff5" member_type: VOTER last_known_addr { host: "127.3.131.190" port: 45897 } }
I20260621 05:11:13.963406 13885 sys_catalog.cc:455] T 00000000000000000000000000000000 P a234a64c229b4e818f0422e3740c8ff5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a234a64c229b4e818f0422e3740c8ff5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a234a64c229b4e818f0422e3740c8ff5" member_type: VOTER last_known_addr { host: "127.3.131.190" port: 45897 } } }
I20260621 05:11:13.963470 13885 sys_catalog.cc:458] T 00000000000000000000000000000000 P a234a64c229b4e818f0422e3740c8ff5 [sys.catalog]: This master's current role is: LEADER
I20260621 05:11:13.963253 13886 sys_catalog.cc:455] T 00000000000000000000000000000000 P a234a64c229b4e818f0422e3740c8ff5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a234a64c229b4e818f0422e3740c8ff5. Latest consensus state: current_term: 1 leader_uuid: "a234a64c229b4e818f0422e3740c8ff5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a234a64c229b4e818f0422e3740c8ff5" member_type: VOTER last_known_addr { host: "127.3.131.190" port: 45897 } } }
I20260621 05:11:13.963642 13886 sys_catalog.cc:458] T 00000000000000000000000000000000 P a234a64c229b4e818f0422e3740c8ff5 [sys.catalog]: This master's current role is: LEADER
W20260621 05:11:13.964195 13898 catalog_manager.cc:1558] T 00000000000000000000000000000000 P a234a64c229b4e818f0422e3740c8ff5: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260621 05:11:13.964269 13898 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260621 05:11:13.964316 13899 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260621 05:11:13.964367 13882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:11:13.953856 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47998 (local address 127.3.131.190:45897)
0621 05:11:13.954051 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0621 05:11:13.954059 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0621 05:11:13.954085 (+ 26us) server_negotiation.cc:407] Connection header received
0621 05:11:13.954497 (+ 412us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:11:13.954503 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:11:13.954551 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:11:13.954708 (+ 157us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:11:13.955460 (+ 752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:11:13.956510 (+ 1050us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:11:13.957259 (+ 749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:11:13.957698 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:11:13.957826 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:11:13.960587 (+ 2761us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:11:13.960592 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:11:13.960592 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:11:13.960610 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:11:13.962588 (+ 1978us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:11:13.962990 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:11:13.962993 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:11:13.962994 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:11:13.963104 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:11:13.963337 (+ 233us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:11:13.963338 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:11:13.963338 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:11:13.963691 (+ 353us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:11:13.963842 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0621 05:11:13.963963 (+ 121us) server_negotiation.cc:299] Negotiation successful
0621 05:11:13.964109 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":144,"thread_start_us":88,"threads_started":1}
I20260621 05:11:13.964466 13899 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260621 05:11:13.966070 13899 catalog_manager.cc:1347] Generated new cluster ID: 23e62ee91e8345ef83de40ba78c1909d
I20260621 05:11:13.966130 13899 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260621 05:11:13.977813 13899 catalog_manager.cc:1370] Generated new certificate authority record
I20260621 05:11:13.978369 13899 catalog_manager.cc:1504] Loading token signing keys...
I20260621 05:11:13.983428 13899 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a234a64c229b4e818f0422e3740c8ff5: Generated new TSK 0
I20260621 05:11:13.983667 13899 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.3.131.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.131.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.131.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782018616648578-3598-0/krb5kdc/kudu_127.3.131.129.keytab.
Entry for principal kudu/127.3.131.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782018616648578-3598-0/krb5kdc/kudu_127.3.131.129.keytab.
I20260621 05:11:14.015455 3598 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWZOx2_/build/release/bin/kudu
/tmp/dist-test-taskWZOx2_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782018616648578-3598-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782018616648578-3598-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782018616648578-3598-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782018616648578-3598-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.131.129:0
--local_ip_for_outbound_sockets=127.3.131.129
--webserver_interface=127.3.131.129
--webserver_port=0
--tserver_master_addrs=127.3.131.190:45897
--keytab_file=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782018616648578-3598-0/krb5kdc/kudu_127.3.131.129.keytab
--principal=kudu/127.3.131.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.131.148:37439
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782018616648578-3598-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782018616648578-3598-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782018616648578-3598-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 05:11:14.103991 13905 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 05:11:14.104207 13905 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 05:11:14.104231 13905 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 05:11:14.105392 13905 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 05:11:14.105435 13905 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 05:11:14.105461 13905 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.131.129
I20260621 05:11:14.106948 13905 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.131.148:37439
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782018616648578-3598-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782018616648578-3598-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782018616648578-3598-0/krb5kdc/kudu_127.3.131.129.keytab
--principal=kudu/127.3.131.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.131.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782018616648578-3598-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.131.129
--webserver_port=0
--tserver_master_addrs=127.3.131.190:45897
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13905
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.131.129
--log_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782018616648578-3598-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 21 Jun 2026 04:16:16 UTC on ccycloud.quasar-xycaha.root.comops.site
build id 4457881
I20260621 05:11:14.107218 13905 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 05:11:14.107465 13905 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 05:11:14.112602 13913 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:11:14.112605 13910 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:11:14.112679 13911 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:11:14.112864 13905 server_base.cc:1034] running on GCE node
I20260621 05:11:14.113093 13905 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 05:11:14.113327 13905 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 05:11:14.114482 13905 hybrid_clock.cc:648] HybridClock initialized: now 1782018674114485 us; error 45 us; skew 500 ppm
Jun 21 05:11:14 dist-test-slave-vrq1 krb5kdc[13798](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782018674, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.131.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 05:11:14.117393 13905 init.cc:377] Logged in from keytab as kudu/127.3.131.129@KRBTEST.COM (short username kudu)
I20260621 05:11:14.117883 13905 webserver.cc:460] Webserver started at http://127.3.131.129:37755/ using document root <none> and password file <none>
I20260621 05:11:14.118131 13905 fs_manager.cc:359] Metadata directory not provided
I20260621 05:11:14.118201 13905 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 05:11:14.118342 13905 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 05:11:14.119521 13905 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782018616648578-3598-0/minicluster-data/ts-0/data/instance:
uuid: "02aa1e50c2f14333bcc546ee8be8e663"
format_stamp: "Formatted at 2026-06-21 05:11:14 on dist-test-slave-vrq1"
server_key: "c9799a5f5369853a8ff529d1f3301d98"
server_key_iv: "d9a3d5d40e89121449b7c1fdfc02dc54"
server_key_version: "encryptionkey@0"
I20260621 05:11:14.119902 13905 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782018616648578-3598-0/minicluster-data/ts-0/wal/instance:
uuid: "02aa1e50c2f14333bcc546ee8be8e663"
format_stamp: "Formatted at 2026-06-21 05:11:14 on dist-test-slave-vrq1"
server_key: "c9799a5f5369853a8ff529d1f3301d98"
server_key_iv: "d9a3d5d40e89121449b7c1fdfc02dc54"
server_key_version: "encryptionkey@0"
I20260621 05:11:14.121335 13905 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260621 05:11:14.122236 13920 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 05:11:14.122465 13905 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260621 05:11:14.122571 13905 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782018616648578-3598-0/minicluster-data/ts-0/data,/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782018616648578-3598-0/minicluster-data/ts-0/wal
uuid: "02aa1e50c2f14333bcc546ee8be8e663"
format_stamp: "Formatted at 2026-06-21 05:11:14 on dist-test-slave-vrq1"
server_key: "c9799a5f5369853a8ff529d1f3301d98"
server_key_iv: "d9a3d5d40e89121449b7c1fdfc02dc54"
server_key_version: "encryptionkey@0"
I20260621 05:11:14.122663 13905 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782018616648578-3598-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782018616648578-3598-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782018616648578-3598-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 05:11:14.134379 13905 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 05:11:14.136797 13905 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 05:11:14.136976 13905 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 05:11:14.137223 13905 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 05:11:14.137557 13905 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 05:11:14.137609 13905 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 05:11:14.137660 13905 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 05:11:14.137688 13905 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 05:11:14.143762 13905 rpc_server.cc:307] RPC server started. Bound to: 127.3.131.129:42489
I20260621 05:11:14.143813 14033 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.131.129:42489 every 8 connection(s)
I20260621 05:11:14.144151 13905 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782018616648578-3598-0/minicluster-data/ts-0/data/info.pb
Jun 21 05:11:14 dist-test-slave-vrq1 krb5kdc[13798](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782018674, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.131.129@KRBTEST.COM for kudu/127.3.131.190@KRBTEST.COM
I20260621 05:11:14.150914 3598 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWZOx2_/build/release/bin/kudu as pid 13905
I20260621 05:11:14.151028 3598 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782018616648578-3598-0/minicluster-data/ts-0/wal/instance
I20260621 05:11:14.151315 3598 external_mini_cluster.cc:1428] Setting key e353b0757943af10a5df03fbd91a37b2
I20260621 05:11:14.154182 14036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:11:14.144907 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.131.190:45897 (local address 127.3.131.129:43973)
0621 05:11:14.145121 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0621 05:11:14.145163 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0621 05:11:14.145644 (+ 481us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:11:14.145853 (+ 209us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:11:14.145857 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:11:14.146439 (+ 582us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:11:14.146710 (+ 271us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:11:14.146717 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:11:14.147445 (+ 728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:11:14.147445 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:11:14.148273 (+ 828us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:11:14.148275 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:11:14.148631 (+ 356us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:11:14.148632 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:11:14.148738 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:11:14.149234 (+ 496us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 05:11:14.149243 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:11:14.150635 (+ 1392us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 05:11:14.152438 (+ 1803us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:11:14.152447 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:11:14.152450 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:11:14.152818 (+ 368us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:11:14.153010 (+ 192us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:11:14.153011 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:11:14.153011 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:11:14.153175 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:11:14.153469 (+ 294us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 05:11:14.153476 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 05:11:14.153685 (+ 209us) client_negotiation.cc:769] Sending connection context
0621 05:11:14.153815 (+ 130us) client_negotiation.cc:240] Negotiation successful
0621 05:11:14.154008 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":98,"threads_started":1}
I20260621 05:11:14.154191 13882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:11:14.144923 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.131.129:43973 (local address 127.3.131.190:45897)
0621 05:11:14.144986 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0621 05:11:14.144989 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0621 05:11:14.145247 (+ 258us) server_negotiation.cc:407] Connection header received
0621 05:11:14.145739 (+ 492us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:11:14.145740 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:11:14.145751 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:11:14.145794 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:11:14.146814 (+ 1020us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:11:14.147361 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:11:14.148355 (+ 994us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:11:14.148554 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:11:14.148617 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:11:14.150720 (+ 2103us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:11:14.150721 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:11:14.150722 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:11:14.150729 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:11:14.152362 (+ 1633us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:11:14.152896 (+ 534us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:11:14.152897 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:11:14.152897 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:11:14.152948 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:11:14.153262 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:11:14.153263 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:11:14.153263 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:11:14.153394 (+ 131us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:11:14.153465 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0621 05:11:14.153973 (+ 508us) server_negotiation.cc:299] Negotiation successful
0621 05:11:14.154084 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260621 05:11:14.154801 14034 heartbeater.cc:347] Connected to a master server at 127.3.131.190:45897
I20260621 05:11:14.154912 14034 heartbeater.cc:464] Registering TS with master...
I20260621 05:11:14.155129 14034 heartbeater.cc:511] Master 127.3.131.190:45897 requested a full tablet report, sending...
I20260621 05:11:14.155752 13843 ts_manager.cc:194] Registered new tserver with Master: 02aa1e50c2f14333bcc546ee8be8e663 (127.3.131.129:42489)
I20260621 05:11:14.156719 13843 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.131.129@KRBTEST.COM'} at 127.3.131.129:43973
WARNING: no policy specified for kudu/127.3.131.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.131.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.131.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782018616648578-3598-0/krb5kdc/kudu_127.3.131.130.keytab.
Entry for principal kudu/127.3.131.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782018616648578-3598-0/krb5kdc/kudu_127.3.131.130.keytab.
I20260621 05:11:14.183210 3598 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWZOx2_/build/release/bin/kudu
/tmp/dist-test-taskWZOx2_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782018616648578-3598-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782018616648578-3598-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782018616648578-3598-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782018616648578-3598-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.131.130:0
--local_ip_for_outbound_sockets=127.3.131.130
--webserver_interface=127.3.131.130
--webserver_port=0
--tserver_master_addrs=127.3.131.190:45897
--keytab_file=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782018616648578-3598-0/krb5kdc/kudu_127.3.131.130.keytab
--principal=kudu/127.3.131.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.131.148:37439
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782018616648578-3598-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782018616648578-3598-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782018616648578-3598-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 05:11:14.267663 14039 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 05:11:14.267882 14039 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 05:11:14.267908 14039 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 05:11:14.269229 14039 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 05:11:14.269280 14039 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 05:11:14.269308 14039 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.131.130
I20260621 05:11:14.270819 14039 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.131.148:37439
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782018616648578-3598-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782018616648578-3598-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782018616648578-3598-0/krb5kdc/kudu_127.3.131.130.keytab
--principal=kudu/127.3.131.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.131.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782018616648578-3598-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.131.130
--webserver_port=0
--tserver_master_addrs=127.3.131.190:45897
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14039
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.131.130
--log_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782018616648578-3598-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 21 Jun 2026 04:16:16 UTC on ccycloud.quasar-xycaha.root.comops.site
build id 4457881
I20260621 05:11:14.271030 14039 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 05:11:14.271267 14039 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 05:11:14.276253 14045 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:11:14.276249 14044 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:11:14.276286 14047 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:11:14.276561 14039 server_base.cc:1034] running on GCE node
I20260621 05:11:14.276762 14039 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 05:11:14.276998 14039 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 05:11:14.278179 14039 hybrid_clock.cc:648] HybridClock initialized: now 1782018674278160 us; error 40 us; skew 500 ppm
Jun 21 05:11:14 dist-test-slave-vrq1 krb5kdc[13798](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782018674, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.131.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 05:11:14.281054 14039 init.cc:377] Logged in from keytab as kudu/127.3.131.130@KRBTEST.COM (short username kudu)
I20260621 05:11:14.281617 14039 webserver.cc:460] Webserver started at http://127.3.131.130:38099/ using document root <none> and password file <none>
I20260621 05:11:14.281862 14039 fs_manager.cc:359] Metadata directory not provided
I20260621 05:11:14.281930 14039 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 05:11:14.282078 14039 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 05:11:14.283288 14039 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782018616648578-3598-0/minicluster-data/ts-1/data/instance:
uuid: "01ab7da76ff64100b96b1cd6f9312b8c"
format_stamp: "Formatted at 2026-06-21 05:11:14 on dist-test-slave-vrq1"
server_key: "4bee36012c8cf9262cf19a526019c972"
server_key_iv: "fdfa3c8910277009b10d692bb1d4605b"
server_key_version: "encryptionkey@0"
I20260621 05:11:14.283667 14039 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782018616648578-3598-0/minicluster-data/ts-1/wal/instance:
uuid: "01ab7da76ff64100b96b1cd6f9312b8c"
format_stamp: "Formatted at 2026-06-21 05:11:14 on dist-test-slave-vrq1"
server_key: "4bee36012c8cf9262cf19a526019c972"
server_key_iv: "fdfa3c8910277009b10d692bb1d4605b"
server_key_version: "encryptionkey@0"
I20260621 05:11:14.285107 14039 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260621 05:11:14.285974 14054 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 05:11:14.286182 14039 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260621 05:11:14.286293 14039 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782018616648578-3598-0/minicluster-data/ts-1/data,/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782018616648578-3598-0/minicluster-data/ts-1/wal
uuid: "01ab7da76ff64100b96b1cd6f9312b8c"
format_stamp: "Formatted at 2026-06-21 05:11:14 on dist-test-slave-vrq1"
server_key: "4bee36012c8cf9262cf19a526019c972"
server_key_iv: "fdfa3c8910277009b10d692bb1d4605b"
server_key_version: "encryptionkey@0"
I20260621 05:11:14.286367 14039 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782018616648578-3598-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782018616648578-3598-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782018616648578-3598-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 05:11:14.294986 14039 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 05:11:14.297313 14039 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 05:11:14.297475 14039 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 05:11:14.297695 14039 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 05:11:14.297994 14039 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 05:11:14.298043 14039 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 05:11:14.298086 14039 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 05:11:14.298122 14039 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 05:11:14.303894 14039 rpc_server.cc:307] RPC server started. Bound to: 127.3.131.130:40249
I20260621 05:11:14.303972 14167 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.131.130:40249 every 8 connection(s)
I20260621 05:11:14.304291 14039 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782018616648578-3598-0/minicluster-data/ts-1/data/info.pb
I20260621 05:11:14.308686 3598 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWZOx2_/build/release/bin/kudu as pid 14039
I20260621 05:11:14.308794 3598 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782018616648578-3598-0/minicluster-data/ts-1/wal/instance
I20260621 05:11:14.309024 3598 external_mini_cluster.cc:1428] Setting key 61c41c2b06a6d30c06dbb0784a33e358
Jun 21 05:11:14 dist-test-slave-vrq1 krb5kdc[13798](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782018674, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.131.130@KRBTEST.COM for kudu/127.3.131.190@KRBTEST.COM
I20260621 05:11:14.315711 13882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:11:14.305044 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.131.130:40369 (local address 127.3.131.190:45897)
0621 05:11:14.305123 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0621 05:11:14.305125 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:11:14.305313 (+ 188us) server_negotiation.cc:407] Connection header received
0621 05:11:14.305873 (+ 560us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:11:14.305875 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:11:14.305886 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:11:14.305951 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:11:14.307009 (+ 1058us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:11:14.307606 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:11:14.308618 (+ 1012us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:11:14.308955 (+ 337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:11:14.309027 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:11:14.311905 (+ 2878us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:11:14.311907 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:11:14.311907 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:11:14.311914 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:11:14.313805 (+ 1891us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:11:14.314387 (+ 582us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:11:14.314389 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:11:14.314389 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:11:14.314458 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:11:14.314841 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:11:14.314842 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:11:14.314842 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:11:14.315019 (+ 177us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:11:14.315111 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0621 05:11:14.315452 (+ 341us) server_negotiation.cc:299] Negotiation successful
0621 05:11:14.315553 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260621 05:11:14.315747 14170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:11:14.305012 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.131.190:45897 (local address 127.3.131.130:40369)
0621 05:11:14.305180 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0621 05:11:14.305224 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0621 05:11:14.305741 (+ 517us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:11:14.306069 (+ 328us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:11:14.306073 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:11:14.306650 (+ 577us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:11:14.306845 (+ 195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:11:14.306850 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:11:14.307705 (+ 855us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:11:14.307705 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:11:14.308529 (+ 824us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:11:14.308532 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:11:14.309120 (+ 588us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:11:14.309121 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:11:14.309251 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:11:14.309895 (+ 644us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 05:11:14.309906 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:11:14.311791 (+ 1885us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 05:11:14.314042 (+ 2251us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:11:14.314050 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:11:14.314052 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:11:14.314308 (+ 256us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:11:14.314638 (+ 330us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:11:14.314640 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:11:14.314640 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:11:14.314765 (+ 125us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:11:14.315108 (+ 343us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 05:11:14.315114 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 05:11:14.315313 (+ 199us) client_negotiation.cc:769] Sending connection context
0621 05:11:14.315408 (+ 95us) client_negotiation.cc:240] Negotiation successful
0621 05:11:14.315560 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":102,"threads_started":1}
I20260621 05:11:14.316370 14168 heartbeater.cc:347] Connected to a master server at 127.3.131.190:45897
I20260621 05:11:14.316511 14168 heartbeater.cc:464] Registering TS with master...
I20260621 05:11:14.316736 14168 heartbeater.cc:511] Master 127.3.131.190:45897 requested a full tablet report, sending...
I20260621 05:11:14.317204 13843 ts_manager.cc:194] Registered new tserver with Master: 01ab7da76ff64100b96b1cd6f9312b8c (127.3.131.130:40249)
I20260621 05:11:14.317742 13843 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.131.130@KRBTEST.COM'} at 127.3.131.130:40369
WARNING: no policy specified for kudu/127.3.131.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.131.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.131.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782018616648578-3598-0/krb5kdc/kudu_127.3.131.131.keytab.
Entry for principal kudu/127.3.131.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782018616648578-3598-0/krb5kdc/kudu_127.3.131.131.keytab.
I20260621 05:11:14.338635 3598 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWZOx2_/build/release/bin/kudu
/tmp/dist-test-taskWZOx2_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782018616648578-3598-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782018616648578-3598-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782018616648578-3598-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782018616648578-3598-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.131.131:0
--local_ip_for_outbound_sockets=127.3.131.131
--webserver_interface=127.3.131.131
--webserver_port=0
--tserver_master_addrs=127.3.131.190:45897
--keytab_file=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782018616648578-3598-0/krb5kdc/kudu_127.3.131.131.keytab
--principal=kudu/127.3.131.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.131.148:37439
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782018616648578-3598-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782018616648578-3598-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782018616648578-3598-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 05:11:14.425901 14173 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 05:11:14.426131 14173 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 05:11:14.426159 14173 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 05:11:14.427495 14173 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 05:11:14.427541 14173 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 05:11:14.427574 14173 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.131.131
I20260621 05:11:14.429112 14173 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.131.148:37439
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782018616648578-3598-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782018616648578-3598-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782018616648578-3598-0/krb5kdc/kudu_127.3.131.131.keytab
--principal=kudu/127.3.131.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.131.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782018616648578-3598-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.131.131
--webserver_port=0
--tserver_master_addrs=127.3.131.190:45897
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14173
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.131.131
--log_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782018616648578-3598-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 21 Jun 2026 04:16:16 UTC on ccycloud.quasar-xycaha.root.comops.site
build id 4457881
I20260621 05:11:14.429330 14173 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 05:11:14.429518 14173 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 05:11:14.434756 14181 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:11:14.434765 14179 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:11:14.434764 14178 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:11:14.435004 14173 server_base.cc:1034] running on GCE node
I20260621 05:11:14.435359 14173 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 05:11:14.435669 14173 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 05:11:14.436849 14173 hybrid_clock.cc:648] HybridClock initialized: now 1782018674436827 us; error 39 us; skew 500 ppm
Jun 21 05:11:14 dist-test-slave-vrq1 krb5kdc[13798](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782018674, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.131.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 05:11:14.439536 14173 init.cc:377] Logged in from keytab as kudu/127.3.131.131@KRBTEST.COM (short username kudu)
I20260621 05:11:14.440008 14173 webserver.cc:460] Webserver started at http://127.3.131.131:35409/ using document root <none> and password file <none>
I20260621 05:11:14.440248 14173 fs_manager.cc:359] Metadata directory not provided
I20260621 05:11:14.440294 14173 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 05:11:14.440429 14173 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 05:11:14.441525 14173 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782018616648578-3598-0/minicluster-data/ts-2/data/instance:
uuid: "e7cffceb63574705847c16b6b2a95db3"
format_stamp: "Formatted at 2026-06-21 05:11:14 on dist-test-slave-vrq1"
server_key: "4ab2c5c264f6cebdc73d5b203221fe4b"
server_key_iv: "1326999c6632abd2c1963d125b1e1165"
server_key_version: "encryptionkey@0"
I20260621 05:11:14.441881 14173 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782018616648578-3598-0/minicluster-data/ts-2/wal/instance:
uuid: "e7cffceb63574705847c16b6b2a95db3"
format_stamp: "Formatted at 2026-06-21 05:11:14 on dist-test-slave-vrq1"
server_key: "4ab2c5c264f6cebdc73d5b203221fe4b"
server_key_iv: "1326999c6632abd2c1963d125b1e1165"
server_key_version: "encryptionkey@0"
I20260621 05:11:14.443321 14173 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260621 05:11:14.444208 14188 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 05:11:14.444439 14173 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260621 05:11:14.444537 14173 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782018616648578-3598-0/minicluster-data/ts-2/data,/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782018616648578-3598-0/minicluster-data/ts-2/wal
uuid: "e7cffceb63574705847c16b6b2a95db3"
format_stamp: "Formatted at 2026-06-21 05:11:14 on dist-test-slave-vrq1"
server_key: "4ab2c5c264f6cebdc73d5b203221fe4b"
server_key_iv: "1326999c6632abd2c1963d125b1e1165"
server_key_version: "encryptionkey@0"
I20260621 05:11:14.444617 14173 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782018616648578-3598-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782018616648578-3598-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782018616648578-3598-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 05:11:14.454319 14173 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 05:11:14.456478 14173 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 05:11:14.456637 14173 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 05:11:14.456864 14173 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 05:11:14.457172 14173 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 05:11:14.457224 14173 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 05:11:14.457265 14173 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 05:11:14.457291 14173 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 05:11:14.463658 14173 rpc_server.cc:307] RPC server started. Bound to: 127.3.131.131:33501
I20260621 05:11:14.463702 14301 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.131.131:33501 every 8 connection(s)
I20260621 05:11:14.464027 14173 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782018616648578-3598-0/minicluster-data/ts-2/data/info.pb
Jun 21 05:11:14 dist-test-slave-vrq1 krb5kdc[13798](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782018674, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.131.131@KRBTEST.COM for kudu/127.3.131.190@KRBTEST.COM
I20260621 05:11:14.473784 13882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:11:14.464775 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.131.131:56461 (local address 127.3.131.190:45897)
0621 05:11:14.464838 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0621 05:11:14.464840 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:11:14.465086 (+ 246us) server_negotiation.cc:407] Connection header received
0621 05:11:14.465557 (+ 471us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:11:14.465558 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:11:14.465568 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:11:14.465656 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:11:14.466594 (+ 938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:11:14.467134 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:11:14.468338 (+ 1204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:11:14.468549 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:11:14.468611 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:11:14.471004 (+ 2393us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:11:14.471006 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:11:14.471007 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:11:14.471014 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:11:14.472379 (+ 1365us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:11:14.472793 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:11:14.472794 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:11:14.472794 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:11:14.472853 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:11:14.473166 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:11:14.473167 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:11:14.473167 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:11:14.473290 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:11:14.473358 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0621 05:11:14.473643 (+ 285us) server_negotiation.cc:299] Negotiation successful
0621 05:11:14.473712 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260621 05:11:14.474061 14304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:11:14.464754 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.131.190:45897 (local address 127.3.131.131:56461)
0621 05:11:14.464905 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0621 05:11:14.464957 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0621 05:11:14.465435 (+ 478us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:11:14.465714 (+ 279us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:11:14.465717 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:11:14.466289 (+ 572us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:11:14.466493 (+ 204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:11:14.466498 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:11:14.467311 (+ 813us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:11:14.467313 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:11:14.468252 (+ 939us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:11:14.468255 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:11:14.468648 (+ 393us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:11:14.468649 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:11:14.468782 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:11:14.469365 (+ 583us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 05:11:14.469374 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:11:14.470857 (+ 1483us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 05:11:14.472461 (+ 1604us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:11:14.472467 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:11:14.472470 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:11:14.472714 (+ 244us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:11:14.472914 (+ 200us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:11:14.472915 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:11:14.472915 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:11:14.473087 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:11:14.473383 (+ 296us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 05:11:14.473389 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 05:11:14.473514 (+ 125us) client_negotiation.cc:769] Sending connection context
0621 05:11:14.473637 (+ 123us) client_negotiation.cc:240] Negotiation successful
0621 05:11:14.473769 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":82,"threads_started":1}
I20260621 05:11:14.474180 3598 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWZOx2_/build/release/bin/kudu as pid 14173
I20260621 05:11:14.474277 3598 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782018616648578-3598-0/minicluster-data/ts-2/wal/instance
I20260621 05:11:14.474489 3598 external_mini_cluster.cc:1428] Setting key 6098efe84edce497ed17710a180bd461
I20260621 05:11:14.474608 14302 heartbeater.cc:347] Connected to a master server at 127.3.131.190:45897
I20260621 05:11:14.474725 14302 heartbeater.cc:464] Registering TS with master...
I20260621 05:11:14.474954 14302 heartbeater.cc:511] Master 127.3.131.190:45897 requested a full tablet report, sending...
I20260621 05:11:14.475361 13843 ts_manager.cc:194] Registered new tserver with Master: e7cffceb63574705847c16b6b2a95db3 (127.3.131.131:33501)
I20260621 05:11:14.475765 13843 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.131.131@KRBTEST.COM'} at 127.3.131.131:56461
I20260621 05:11:14.486644 3598 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260621 05:11:14.494520 13882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:11:14.488076 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48002 (local address 127.3.131.190:45897)
0621 05:11:14.488154 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0621 05:11:14.488157 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0621 05:11:14.488246 (+ 89us) server_negotiation.cc:407] Connection header received
0621 05:11:14.488289 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:11:14.488291 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:11:14.488300 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:11:14.488350 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:11:14.488873 (+ 523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:11:14.489486 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:11:14.490245 (+ 759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:11:14.490517 (+ 272us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:11:14.490571 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:11:14.491829 (+ 1258us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:11:14.491831 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:11:14.491831 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:11:14.491838 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:11:14.493082 (+ 1244us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:11:14.493426 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:11:14.493426 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:11:14.493427 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:11:14.493483 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:11:14.493754 (+ 271us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:11:14.493756 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:11:14.493756 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:11:14.493919 (+ 163us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:11:14.494015 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0621 05:11:14.494199 (+ 184us) server_negotiation.cc:299] Negotiation successful
0621 05:11:14.494268 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260621 05:11:14.496340 13843 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48002:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260621 05:11:14.497888 13843 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260621 05:11:14.504384 14315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:11:14.500327 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43204 (local address 127.3.131.131:33501)
0621 05:11:14.500498 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0621 05:11:14.500504 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 05:11:14.500516 (+ 12us) server_negotiation.cc:407] Connection header received
0621 05:11:14.500805 (+ 289us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:11:14.500808 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:11:14.500822 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:11:14.500893 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 05:11:14.501112 (+ 219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:11:14.502255 (+ 1143us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:11:14.503406 (+ 1151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:11:14.503993 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:11:14.504079 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:11:14.504151 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0621 05:11:14.504255 (+ 104us) server_negotiation.cc:299] Negotiation successful
0621 05:11:14.504297 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":137,"thread_start_us":99,"threads_started":1}
I20260621 05:11:14.504410 14314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:11:14.500201 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.131.131:33501 (local address 127.0.0.1:43204)
0621 05:11:14.500355 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0621 05:11:14.500418 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0621 05:11:14.500729 (+ 311us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:11:14.500974 (+ 245us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:11:14.500975 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:11:14.500978 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 05:11:14.501028 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:11:14.501030 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:11:14.502375 (+ 1345us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:11:14.502375 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:11:14.503324 (+ 949us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:11:14.503326 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:11:14.504077 (+ 751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:11:14.504078 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:11:14.504153 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:11:14.504168 (+ 15us) client_negotiation.cc:769] Sending connection context
0621 05:11:14.504241 (+ 73us) client_negotiation.cc:240] Negotiation successful
0621 05:11:14.504295 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":118,"mutex_wait_us":30,"thread_start_us":55,"threads_started":1}
I20260621 05:11:14.504717 14313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:11:14.500460 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.131.129:42489 (local address 127.0.0.1:54894)
0621 05:11:14.500592 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0621 05:11:14.500602 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0621 05:11:14.500681 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:11:14.501097 (+ 416us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:11:14.501098 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:11:14.501100 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 05:11:14.501151 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:11:14.501152 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:11:14.502381 (+ 1229us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:11:14.502382 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:11:14.503538 (+ 1156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:11:14.503541 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:11:14.504486 (+ 945us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:11:14.504487 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:11:14.504573 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:11:14.504577 (+ 4us) client_negotiation.cc:769] Sending connection context
0621 05:11:14.504625 (+ 48us) client_negotiation.cc:240] Negotiation successful
0621 05:11:14.504657 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121,"mutex_wait_us":27,"thread_start_us":85,"threads_started":1}
I20260621 05:11:14.504724 14317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:11:14.500547 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47032 (local address 127.3.131.130:40249)
0621 05:11:14.500734 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0621 05:11:14.500739 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 05:11:14.500748 (+ 9us) server_negotiation.cc:407] Connection header received
0621 05:11:14.500786 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:11:14.500787 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:11:14.500804 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:11:14.500890 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 05:11:14.501130 (+ 240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:11:14.501862 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:11:14.503300 (+ 1438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:11:14.504028 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:11:14.504098 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:11:14.504149 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0621 05:11:14.504592 (+ 443us) server_negotiation.cc:299] Negotiation successful
0621 05:11:14.504644 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":159,"thread_start_us":83,"threads_started":1}
I20260621 05:11:14.504979 14318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:11:14.500586 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54894 (local address 127.3.131.129:42489)
0621 05:11:14.500849 (+ 263us) server_negotiation.cc:206] Beginning negotiation
0621 05:11:14.500852 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0621 05:11:14.500877 (+ 25us) server_negotiation.cc:407] Connection header received
0621 05:11:14.500913 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:11:14.500915 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:11:14.500930 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:11:14.501009 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 05:11:14.501230 (+ 221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:11:14.502255 (+ 1025us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:11:14.503615 (+ 1360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:11:14.504335 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:11:14.504393 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:11:14.504454 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0621 05:11:14.504839 (+ 385us) server_negotiation.cc:299] Negotiation successful
0621 05:11:14.504895 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":235,"thread_start_us":78,"threads_started":1}
I20260621 05:11:14.505177 14236 tablet_service.cc:1467] Processing CreateTablet for tablet 7118043ce2504ccc854009f5293f0ace (DEFAULT_TABLE table=test-table [id=9d06f49923ed4931be2930a84e266d13]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 05:11:14.505556 14236 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7118043ce2504ccc854009f5293f0ace. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 05:11:14.505599 13968 tablet_service.cc:1467] Processing CreateTablet for tablet 7118043ce2504ccc854009f5293f0ace (DEFAULT_TABLE table=test-table [id=9d06f49923ed4931be2930a84e266d13]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 05:11:14.505896 13968 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7118043ce2504ccc854009f5293f0ace. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 05:11:14.507321 14316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:11:14.500217 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.131.130:40249 (local address 127.0.0.1:47032)
0621 05:11:14.500583 (+ 366us) negotiation.cc:107] Waiting for socket to connect
0621 05:11:14.500595 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0621 05:11:14.500662 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:11:14.500906 (+ 244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:11:14.500909 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:11:14.500911 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 05:11:14.500988 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:11:14.500990 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:11:14.501965 (+ 975us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:11:14.501966 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:11:14.503215 (+ 1249us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:11:14.503217 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:11:14.504184 (+ 967us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:11:14.504185 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:11:14.504247 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:11:14.504251 (+ 4us) client_negotiation.cc:769] Sending connection context
0621 05:11:14.504299 (+ 48us) client_negotiation.cc:240] Negotiation successful
0621 05:11:14.504345 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":347,"thread_start_us":86,"threads_started":1}
I20260621 05:11:14.508051 14102 tablet_service.cc:1467] Processing CreateTablet for tablet 7118043ce2504ccc854009f5293f0ace (DEFAULT_TABLE table=test-table [id=9d06f49923ed4931be2930a84e266d13]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 05:11:14.508397 14102 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7118043ce2504ccc854009f5293f0ace. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 05:11:14.509075 14320 tablet_bootstrap.cc:492] T 7118043ce2504ccc854009f5293f0ace P 02aa1e50c2f14333bcc546ee8be8e663: Bootstrap starting.
I20260621 05:11:14.509281 14319 tablet_bootstrap.cc:492] T 7118043ce2504ccc854009f5293f0ace P e7cffceb63574705847c16b6b2a95db3: Bootstrap starting.
I20260621 05:11:14.510015 14319 tablet_bootstrap.cc:654] T 7118043ce2504ccc854009f5293f0ace P e7cffceb63574705847c16b6b2a95db3: Neither blocks nor log segments found. Creating new log.
I20260621 05:11:14.510598 14319 log.cc:824] T 7118043ce2504ccc854009f5293f0ace P e7cffceb63574705847c16b6b2a95db3: Log is configured to *not* fsync() on all Append() calls
I20260621 05:11:14.511426 14319 tablet_bootstrap.cc:492] T 7118043ce2504ccc854009f5293f0ace P e7cffceb63574705847c16b6b2a95db3: No bootstrap required, opened a new log
I20260621 05:11:14.511514 14319 ts_tablet_manager.cc:1397] T 7118043ce2504ccc854009f5293f0ace P e7cffceb63574705847c16b6b2a95db3: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260621 05:11:14.511588 14320 tablet_bootstrap.cc:654] T 7118043ce2504ccc854009f5293f0ace P 02aa1e50c2f14333bcc546ee8be8e663: Neither blocks nor log segments found. Creating new log.
I20260621 05:11:14.512055 14322 tablet_bootstrap.cc:492] T 7118043ce2504ccc854009f5293f0ace P 01ab7da76ff64100b96b1cd6f9312b8c: Bootstrap starting.
I20260621 05:11:14.512161 14320 log.cc:824] T 7118043ce2504ccc854009f5293f0ace P 02aa1e50c2f14333bcc546ee8be8e663: Log is configured to *not* fsync() on all Append() calls
I20260621 05:11:14.512710 14322 tablet_bootstrap.cc:654] T 7118043ce2504ccc854009f5293f0ace P 01ab7da76ff64100b96b1cd6f9312b8c: Neither blocks nor log segments found. Creating new log.
I20260621 05:11:14.513022 14320 tablet_bootstrap.cc:492] T 7118043ce2504ccc854009f5293f0ace P 02aa1e50c2f14333bcc546ee8be8e663: No bootstrap required, opened a new log
I20260621 05:11:14.513137 14320 ts_tablet_manager.cc:1397] T 7118043ce2504ccc854009f5293f0ace P 02aa1e50c2f14333bcc546ee8be8e663: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.002s
I20260621 05:11:14.513286 14322 log.cc:824] T 7118043ce2504ccc854009f5293f0ace P 01ab7da76ff64100b96b1cd6f9312b8c: Log is configured to *not* fsync() on all Append() calls
I20260621 05:11:14.513710 14319 raft_consensus.cc:348] T 7118043ce2504ccc854009f5293f0ace P e7cffceb63574705847c16b6b2a95db3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7cffceb63574705847c16b6b2a95db3" member_type: VOTER last_known_addr { host: "127.3.131.131" port: 33501 } } peers { permanent_uuid: "02aa1e50c2f14333bcc546ee8be8e663" member_type: VOTER last_known_addr { host: "127.3.131.129" port: 42489 } } peers { permanent_uuid: "01ab7da76ff64100b96b1cd6f9312b8c" member_type: VOTER last_known_addr { host: "127.3.131.130" port: 40249 } }
I20260621 05:11:14.513899 14319 raft_consensus.cc:374] T 7118043ce2504ccc854009f5293f0ace P e7cffceb63574705847c16b6b2a95db3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 05:11:14.513943 14319 raft_consensus.cc:729] T 7118043ce2504ccc854009f5293f0ace P e7cffceb63574705847c16b6b2a95db3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e7cffceb63574705847c16b6b2a95db3, State: Initialized, Role: FOLLOWER
I20260621 05:11:14.514104 14322 tablet_bootstrap.cc:492] T 7118043ce2504ccc854009f5293f0ace P 01ab7da76ff64100b96b1cd6f9312b8c: No bootstrap required, opened a new log
I20260621 05:11:14.514091 14319 consensus_queue.cc:260] T 7118043ce2504ccc854009f5293f0ace P e7cffceb63574705847c16b6b2a95db3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7cffceb63574705847c16b6b2a95db3" member_type: VOTER last_known_addr { host: "127.3.131.131" port: 33501 } } peers { permanent_uuid: "02aa1e50c2f14333bcc546ee8be8e663" member_type: VOTER last_known_addr { host: "127.3.131.129" port: 42489 } } peers { permanent_uuid: "01ab7da76ff64100b96b1cd6f9312b8c" member_type: VOTER last_known_addr { host: "127.3.131.130" port: 40249 } }
I20260621 05:11:14.514179 14322 ts_tablet_manager.cc:1397] T 7118043ce2504ccc854009f5293f0ace P 01ab7da76ff64100b96b1cd6f9312b8c: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260621 05:11:14.514369 14319 ts_tablet_manager.cc:1428] T 7118043ce2504ccc854009f5293f0ace P e7cffceb63574705847c16b6b2a95db3: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260621 05:11:14.514497 14302 heartbeater.cc:503] Master 127.3.131.190:45897 was elected leader, sending a full tablet report...
I20260621 05:11:14.515316 14320 raft_consensus.cc:348] T 7118043ce2504ccc854009f5293f0ace P 02aa1e50c2f14333bcc546ee8be8e663 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7cffceb63574705847c16b6b2a95db3" member_type: VOTER last_known_addr { host: "127.3.131.131" port: 33501 } } peers { permanent_uuid: "02aa1e50c2f14333bcc546ee8be8e663" member_type: VOTER last_known_addr { host: "127.3.131.129" port: 42489 } } peers { permanent_uuid: "01ab7da76ff64100b96b1cd6f9312b8c" member_type: VOTER last_known_addr { host: "127.3.131.130" port: 40249 } }
I20260621 05:11:14.515496 14320 raft_consensus.cc:374] T 7118043ce2504ccc854009f5293f0ace P 02aa1e50c2f14333bcc546ee8be8e663 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 05:11:14.515530 14320 raft_consensus.cc:729] T 7118043ce2504ccc854009f5293f0ace P 02aa1e50c2f14333bcc546ee8be8e663 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 02aa1e50c2f14333bcc546ee8be8e663, State: Initialized, Role: FOLLOWER
I20260621 05:11:14.515661 14320 consensus_queue.cc:260] T 7118043ce2504ccc854009f5293f0ace P 02aa1e50c2f14333bcc546ee8be8e663 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7cffceb63574705847c16b6b2a95db3" member_type: VOTER last_known_addr { host: "127.3.131.131" port: 33501 } } peers { permanent_uuid: "02aa1e50c2f14333bcc546ee8be8e663" member_type: VOTER last_known_addr { host: "127.3.131.129" port: 42489 } } peers { permanent_uuid: "01ab7da76ff64100b96b1cd6f9312b8c" member_type: VOTER last_known_addr { host: "127.3.131.130" port: 40249 } }
I20260621 05:11:14.515951 14320 ts_tablet_manager.cc:1428] T 7118043ce2504ccc854009f5293f0ace P 02aa1e50c2f14333bcc546ee8be8e663: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
I20260621 05:11:14.516085 14034 heartbeater.cc:503] Master 127.3.131.190:45897 was elected leader, sending a full tablet report...
I20260621 05:11:14.516393 14322 raft_consensus.cc:348] T 7118043ce2504ccc854009f5293f0ace P 01ab7da76ff64100b96b1cd6f9312b8c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7cffceb63574705847c16b6b2a95db3" member_type: VOTER last_known_addr { host: "127.3.131.131" port: 33501 } } peers { permanent_uuid: "02aa1e50c2f14333bcc546ee8be8e663" member_type: VOTER last_known_addr { host: "127.3.131.129" port: 42489 } } peers { permanent_uuid: "01ab7da76ff64100b96b1cd6f9312b8c" member_type: VOTER last_known_addr { host: "127.3.131.130" port: 40249 } }
I20260621 05:11:14.516567 14322 raft_consensus.cc:374] T 7118043ce2504ccc854009f5293f0ace P 01ab7da76ff64100b96b1cd6f9312b8c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 05:11:14.516625 14322 raft_consensus.cc:729] T 7118043ce2504ccc854009f5293f0ace P 01ab7da76ff64100b96b1cd6f9312b8c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 01ab7da76ff64100b96b1cd6f9312b8c, State: Initialized, Role: FOLLOWER
I20260621 05:11:14.516743 14322 consensus_queue.cc:260] T 7118043ce2504ccc854009f5293f0ace P 01ab7da76ff64100b96b1cd6f9312b8c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7cffceb63574705847c16b6b2a95db3" member_type: VOTER last_known_addr { host: "127.3.131.131" port: 33501 } } peers { permanent_uuid: "02aa1e50c2f14333bcc546ee8be8e663" member_type: VOTER last_known_addr { host: "127.3.131.129" port: 42489 } } peers { permanent_uuid: "01ab7da76ff64100b96b1cd6f9312b8c" member_type: VOTER last_known_addr { host: "127.3.131.130" port: 40249 } }
I20260621 05:11:14.516960 14322 ts_tablet_manager.cc:1428] T 7118043ce2504ccc854009f5293f0ace P 01ab7da76ff64100b96b1cd6f9312b8c: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260621 05:11:14.517019 14168 heartbeater.cc:503] Master 127.3.131.190:45897 was elected leader, sending a full tablet report...
W20260621 05:11:14.554895 14169 tablet.cc:2369] T 7118043ce2504ccc854009f5293f0ace P 01ab7da76ff64100b96b1cd6f9312b8c: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260621 05:11:14.645054 14035 tablet.cc:2369] T 7118043ce2504ccc854009f5293f0ace P 02aa1e50c2f14333bcc546ee8be8e663: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260621 05:11:14.714677 14303 tablet.cc:2369] T 7118043ce2504ccc854009f5293f0ace P e7cffceb63574705847c16b6b2a95db3: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260621 05:11:14.817875 14327 raft_consensus.cc:482] T 7118043ce2504ccc854009f5293f0ace P 01ab7da76ff64100b96b1cd6f9312b8c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260621 05:11:14.818022 14327 raft_consensus.cc:504] T 7118043ce2504ccc854009f5293f0ace P 01ab7da76ff64100b96b1cd6f9312b8c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7cffceb63574705847c16b6b2a95db3" member_type: VOTER last_known_addr { host: "127.3.131.131" port: 33501 } } peers { permanent_uuid: "02aa1e50c2f14333bcc546ee8be8e663" member_type: VOTER last_known_addr { host: "127.3.131.129" port: 42489 } } peers { permanent_uuid: "01ab7da76ff64100b96b1cd6f9312b8c" member_type: VOTER last_known_addr { host: "127.3.131.130" port: 40249 } }
I20260621 05:11:14.818436 14327 leader_election.cc:288] T 7118043ce2504ccc854009f5293f0ace P 01ab7da76ff64100b96b1cd6f9312b8c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e7cffceb63574705847c16b6b2a95db3 (127.3.131.131:33501), 02aa1e50c2f14333bcc546ee8be8e663 (127.3.131.129:42489)
I20260621 05:11:14.821511 14315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:11:14.818872 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.131.130:52215 (local address 127.3.131.131:33501)
0621 05:11:14.818932 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0621 05:11:14.818934 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:11:14.818945 (+ 11us) server_negotiation.cc:407] Connection header received
0621 05:11:14.819040 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:11:14.819042 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:11:14.819052 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:11:14.819130 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 05:11:14.819369 (+ 239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:11:14.819838 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:11:14.820791 (+ 953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:11:14.821176 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:11:14.821258 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:11:14.821268 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0621 05:11:14.821406 (+ 138us) server_negotiation.cc:299] Negotiation successful
0621 05:11:14.821442 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260621 05:11:14.821532 14328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:11:14.818761 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.131.131:33501 (local address 127.3.131.130:52215)
0621 05:11:14.818879 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0621 05:11:14.818892 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0621 05:11:14.818949 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:11:14.819218 (+ 269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:11:14.819219 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:11:14.819222 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 05:11:14.819288 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:11:14.819290 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:11:14.819923 (+ 633us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:11:14.819924 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:11:14.820696 (+ 772us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:11:14.820698 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:11:14.821256 (+ 558us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:11:14.821257 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:11:14.821333 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:11:14.821336 (+ 3us) client_negotiation.cc:769] Sending connection context
0621 05:11:14.821400 (+ 64us) client_negotiation.cc:240] Negotiation successful
0621 05:11:14.821442 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":99,"thread_start_us":72,"threads_started":1}
I20260621 05:11:14.821645 14318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:11:14.819065 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.131.130:43605 (local address 127.3.131.129:42489)
0621 05:11:14.819135 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0621 05:11:14.819137 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:11:14.819226 (+ 89us) server_negotiation.cc:407] Connection header received
0621 05:11:14.819289 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:11:14.819291 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:11:14.819300 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:11:14.819364 (+ 64us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 05:11:14.819564 (+ 200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:11:14.820026 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:11:14.820945 (+ 919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:11:14.821307 (+ 362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:11:14.821390 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:11:14.821397 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0621 05:11:14.821541 (+ 144us) server_negotiation.cc:299] Negotiation successful
0621 05:11:14.821580 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260621 05:11:14.821655 14329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:11:14.819018 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.131.129:42489 (local address 127.3.131.130:43605)
0621 05:11:14.819139 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0621 05:11:14.819150 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0621 05:11:14.819235 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:11:14.819380 (+ 145us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:11:14.819381 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:11:14.819383 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 05:11:14.819484 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:11:14.819486 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:11:14.820119 (+ 633us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:11:14.820120 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:11:14.820853 (+ 733us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:11:14.820854 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:11:14.821389 (+ 535us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:11:14.821390 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:11:14.821470 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:11:14.821474 (+ 4us) client_negotiation.cc:769] Sending connection context
0621 05:11:14.821536 (+ 62us) client_negotiation.cc:240] Negotiation successful
0621 05:11:14.821574 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":51,"threads_started":1}
I20260621 05:11:14.822104 14256 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7118043ce2504ccc854009f5293f0ace" candidate_uuid: "01ab7da76ff64100b96b1cd6f9312b8c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e7cffceb63574705847c16b6b2a95db3" is_pre_election: true
I20260621 05:11:14.822120 13988 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7118043ce2504ccc854009f5293f0ace" candidate_uuid: "01ab7da76ff64100b96b1cd6f9312b8c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "02aa1e50c2f14333bcc546ee8be8e663" is_pre_election: true
I20260621 05:11:14.822265 14256 raft_consensus.cc:2454] T 7118043ce2504ccc854009f5293f0ace P e7cffceb63574705847c16b6b2a95db3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 01ab7da76ff64100b96b1cd6f9312b8c in term 0.
I20260621 05:11:14.822273 13988 raft_consensus.cc:2454] T 7118043ce2504ccc854009f5293f0ace P 02aa1e50c2f14333bcc546ee8be8e663 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 01ab7da76ff64100b96b1cd6f9312b8c in term 0.
I20260621 05:11:14.822549 14056 leader_election.cc:302] T 7118043ce2504ccc854009f5293f0ace P 01ab7da76ff64100b96b1cd6f9312b8c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 01ab7da76ff64100b96b1cd6f9312b8c, e7cffceb63574705847c16b6b2a95db3; no voters:
I20260621 05:11:14.822674 14327 raft_consensus.cc:2781] T 7118043ce2504ccc854009f5293f0ace P 01ab7da76ff64100b96b1cd6f9312b8c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260621 05:11:14.822726 14327 raft_consensus.cc:482] T 7118043ce2504ccc854009f5293f0ace P 01ab7da76ff64100b96b1cd6f9312b8c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260621 05:11:14.822760 14327 raft_consensus.cc:3037] T 7118043ce2504ccc854009f5293f0ace P 01ab7da76ff64100b96b1cd6f9312b8c [term 0 FOLLOWER]: Advancing to term 1
I20260621 05:11:14.823827 14327 raft_consensus.cc:504] T 7118043ce2504ccc854009f5293f0ace P 01ab7da76ff64100b96b1cd6f9312b8c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7cffceb63574705847c16b6b2a95db3" member_type: VOTER last_known_addr { host: "127.3.131.131" port: 33501 } } peers { permanent_uuid: "02aa1e50c2f14333bcc546ee8be8e663" member_type: VOTER last_known_addr { host: "127.3.131.129" port: 42489 } } peers { permanent_uuid: "01ab7da76ff64100b96b1cd6f9312b8c" member_type: VOTER last_known_addr { host: "127.3.131.130" port: 40249 } }
I20260621 05:11:14.824004 14327 leader_election.cc:288] T 7118043ce2504ccc854009f5293f0ace P 01ab7da76ff64100b96b1cd6f9312b8c [CANDIDATE]: Term 1 election: Requested vote from peers e7cffceb63574705847c16b6b2a95db3 (127.3.131.131:33501), 02aa1e50c2f14333bcc546ee8be8e663 (127.3.131.129:42489)
I20260621 05:11:14.824746 13988 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7118043ce2504ccc854009f5293f0ace" candidate_uuid: "01ab7da76ff64100b96b1cd6f9312b8c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "02aa1e50c2f14333bcc546ee8be8e663"
I20260621 05:11:14.824739 14256 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7118043ce2504ccc854009f5293f0ace" candidate_uuid: "01ab7da76ff64100b96b1cd6f9312b8c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e7cffceb63574705847c16b6b2a95db3"
I20260621 05:11:14.824862 13988 raft_consensus.cc:3037] T 7118043ce2504ccc854009f5293f0ace P 02aa1e50c2f14333bcc546ee8be8e663 [term 0 FOLLOWER]: Advancing to term 1
I20260621 05:11:14.824862 14256 raft_consensus.cc:3037] T 7118043ce2504ccc854009f5293f0ace P e7cffceb63574705847c16b6b2a95db3 [term 0 FOLLOWER]: Advancing to term 1
I20260621 05:11:14.826314 14256 raft_consensus.cc:2454] T 7118043ce2504ccc854009f5293f0ace P e7cffceb63574705847c16b6b2a95db3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 01ab7da76ff64100b96b1cd6f9312b8c in term 1.
I20260621 05:11:14.826545 14056 leader_election.cc:302] T 7118043ce2504ccc854009f5293f0ace P 01ab7da76ff64100b96b1cd6f9312b8c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 01ab7da76ff64100b96b1cd6f9312b8c, e7cffceb63574705847c16b6b2a95db3; no voters:
I20260621 05:11:14.826664 14327 raft_consensus.cc:2781] T 7118043ce2504ccc854009f5293f0ace P 01ab7da76ff64100b96b1cd6f9312b8c [term 1 FOLLOWER]: Leader election won for term 1
I20260621 05:11:14.826871 14327 raft_consensus.cc:686] T 7118043ce2504ccc854009f5293f0ace P 01ab7da76ff64100b96b1cd6f9312b8c [term 1 LEADER]: Becoming Leader. State: Replica: 01ab7da76ff64100b96b1cd6f9312b8c, State: Running, Role: LEADER
I20260621 05:11:14.827010 14327 consensus_queue.cc:237] T 7118043ce2504ccc854009f5293f0ace P 01ab7da76ff64100b96b1cd6f9312b8c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7cffceb63574705847c16b6b2a95db3" member_type: VOTER last_known_addr { host: "127.3.131.131" port: 33501 } } peers { permanent_uuid: "02aa1e50c2f14333bcc546ee8be8e663" member_type: VOTER last_known_addr { host: "127.3.131.129" port: 42489 } } peers { permanent_uuid: "01ab7da76ff64100b96b1cd6f9312b8c" member_type: VOTER last_known_addr { host: "127.3.131.130" port: 40249 } }
I20260621 05:11:14.828051 13842 catalog_manager.cc:5495] T 7118043ce2504ccc854009f5293f0ace P 01ab7da76ff64100b96b1cd6f9312b8c reported cstate change: term changed from 0 to 1, leader changed from <none> to 01ab7da76ff64100b96b1cd6f9312b8c (127.3.131.130). New cstate: current_term: 1 leader_uuid: "01ab7da76ff64100b96b1cd6f9312b8c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7cffceb63574705847c16b6b2a95db3" member_type: VOTER last_known_addr { host: "127.3.131.131" port: 33501 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "02aa1e50c2f14333bcc546ee8be8e663" member_type: VOTER last_known_addr { host: "127.3.131.129" port: 42489 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "01ab7da76ff64100b96b1cd6f9312b8c" member_type: VOTER last_known_addr { host: "127.3.131.130" port: 40249 } health_report { overall_health: HEALTHY } } }
I20260621 05:11:14.828428 13988 raft_consensus.cc:2454] T 7118043ce2504ccc854009f5293f0ace P 02aa1e50c2f14333bcc546ee8be8e663 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 01ab7da76ff64100b96b1cd6f9312b8c in term 1.
I20260621 05:11:14.859699 14317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:11:14.856512 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47038 (local address 127.3.131.130:40249)
0621 05:11:14.856560 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0621 05:11:14.856562 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:11:14.856613 (+ 51us) server_negotiation.cc:407] Connection header received
0621 05:11:14.856654 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:11:14.856655 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:11:14.856675 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:11:14.856764 (+ 89us) server_negotiation.cc:226] Negotiated authn=TOKEN
0621 05:11:14.856940 (+ 176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:11:14.857644 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:11:14.858706 (+ 1062us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:11:14.858999 (+ 293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:11:14.859054 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:11:14.859297 (+ 243us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0621 05:11:14.859428 (+ 131us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0621 05:11:14.859501 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0621 05:11:14.859583 (+ 82us) server_negotiation.cc:299] Negotiation successful
0621 05:11:14.859604 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260621 05:11:14.861985 13988 raft_consensus.cc:1261] T 7118043ce2504ccc854009f5293f0ace P 02aa1e50c2f14333bcc546ee8be8e663 [term 1 FOLLOWER]: Refusing update from remote peer 01ab7da76ff64100b96b1cd6f9312b8c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 05:11:14.861985 14256 raft_consensus.cc:1261] T 7118043ce2504ccc854009f5293f0ace P e7cffceb63574705847c16b6b2a95db3 [term 1 FOLLOWER]: Refusing update from remote peer 01ab7da76ff64100b96b1cd6f9312b8c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 05:11:14.862391 14330 consensus_queue.cc:1035] T 7118043ce2504ccc854009f5293f0ace P 01ab7da76ff64100b96b1cd6f9312b8c [LEADER]: Connected to new peer: Peer: permanent_uuid: "02aa1e50c2f14333bcc546ee8be8e663" member_type: VOTER last_known_addr { host: "127.3.131.129" port: 42489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 05:11:14.862530 14327 consensus_queue.cc:1035] T 7118043ce2504ccc854009f5293f0ace P 01ab7da76ff64100b96b1cd6f9312b8c [LEADER]: Connected to new peer: Peer: permanent_uuid: "e7cffceb63574705847c16b6b2a95db3" member_type: VOTER last_known_addr { host: "127.3.131.131" port: 33501 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 05:11:14.866559 14333 mvcc.cc:205] Tried to move back new op lower bound from 7299148492231643136 to 7299148492092362752. Current Snapshot: MvccSnapshot[applied={T|T < 7299148492231643136 or (T in {7299148492231643136})}]
I20260621 05:11:14.866719 14335 mvcc.cc:205] Tried to move back new op lower bound from 7299148492231643136 to 7299148492092362752. Current Snapshot: MvccSnapshot[applied={T|T < 7299148492231643136 or (T in {7299148492231643136})}]
I20260621 05:11:14.867712 13842 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48002:
table { table_name: "test-table" } modify_external_catalogs: true
I20260621 05:11:14.867811 13842 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48002:
table { table_name: "test-table" } modify_external_catalogs: true
I20260621 05:11:14.868505 13842 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a234a64c229b4e818f0422e3740c8ff5: Sending DeleteTablet for 3 replicas of tablet 7118043ce2504ccc854009f5293f0ace
I20260621 05:11:14.868880 14236 tablet_service.cc:1514] Processing DeleteTablet for tablet 7118043ce2504ccc854009f5293f0ace with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 05:11:14 UTC) from {username='kudu', principal='kudu/127.3.131.190@KRBTEST.COM'} at 127.0.0.1:43204
I20260621 05:11:14.868937 13968 tablet_service.cc:1514] Processing DeleteTablet for tablet 7118043ce2504ccc854009f5293f0ace with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 05:11:14 UTC) from {username='kudu', principal='kudu/127.3.131.190@KRBTEST.COM'} at 127.0.0.1:54894
I20260621 05:11:14.869024 14102 tablet_service.cc:1514] Processing DeleteTablet for tablet 7118043ce2504ccc854009f5293f0ace with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 05:11:14 UTC) from {username='kudu', principal='kudu/127.3.131.190@KRBTEST.COM'} at 127.0.0.1:47032
I20260621 05:11:14.869185 14344 tablet_replica.cc:330] T 7118043ce2504ccc854009f5293f0ace P 02aa1e50c2f14333bcc546ee8be8e663: stopping tablet replica
I20260621 05:11:14.869292 14343 tablet_replica.cc:330] T 7118043ce2504ccc854009f5293f0ace P e7cffceb63574705847c16b6b2a95db3: stopping tablet replica
I20260621 05:11:14.869290 14344 raft_consensus.cc:2229] T 7118043ce2504ccc854009f5293f0ace P 02aa1e50c2f14333bcc546ee8be8e663 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260621 05:11:14.869370 14343 raft_consensus.cc:2229] T 7118043ce2504ccc854009f5293f0ace P e7cffceb63574705847c16b6b2a95db3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260621 05:11:14.869400 14344 raft_consensus.cc:2258] T 7118043ce2504ccc854009f5293f0ace P 02aa1e50c2f14333bcc546ee8be8e663 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 05:11:14.869448 14343 raft_consensus.cc:2258] T 7118043ce2504ccc854009f5293f0ace P e7cffceb63574705847c16b6b2a95db3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 05:11:14.869524 3598 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWZOx2_/build/release/bin/kudu with pid 13905
I20260621 05:11:14.871233 14343 ts_tablet_manager.cc:1905] T 7118043ce2504ccc854009f5293f0ace P e7cffceb63574705847c16b6b2a95db3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 05:11:14.872138 14345 tablet_replica.cc:330] T 7118043ce2504ccc854009f5293f0ace P 01ab7da76ff64100b96b1cd6f9312b8c: stopping tablet replica
I20260621 05:11:14.872301 14345 raft_consensus.cc:2229] T 7118043ce2504ccc854009f5293f0ace P 01ab7da76ff64100b96b1cd6f9312b8c [term 1 LEADER]: Raft consensus shutting down.
I20260621 05:11:14.872458 14345 raft_consensus.cc:2258] T 7118043ce2504ccc854009f5293f0ace P 01ab7da76ff64100b96b1cd6f9312b8c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 05:11:14.872968 14345 ts_tablet_manager.cc:1905] T 7118043ce2504ccc854009f5293f0ace P 01ab7da76ff64100b96b1cd6f9312b8c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 05:11:14.873070 14343 ts_tablet_manager.cc:1918] T 7118043ce2504ccc854009f5293f0ace P e7cffceb63574705847c16b6b2a95db3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260621 05:11:14.873147 14343 log.cc:1196] T 7118043ce2504ccc854009f5293f0ace P e7cffceb63574705847c16b6b2a95db3: Deleting WAL directory at /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782018616648578-3598-0/minicluster-data/ts-2/wal/wals/7118043ce2504ccc854009f5293f0ace
I20260621 05:11:14.873421 14343 ts_tablet_manager.cc:1939] T 7118043ce2504ccc854009f5293f0ace P e7cffceb63574705847c16b6b2a95db3: Deleting consensus metadata
I20260621 05:11:14.873853 13828 catalog_manager.cc:4841] TS e7cffceb63574705847c16b6b2a95db3 (127.3.131.131:33501): tablet 7118043ce2504ccc854009f5293f0ace (table test-table [id=9d06f49923ed4931be2930a84e266d13]) successfully deleted
I20260621 05:11:14.875339 14345 ts_tablet_manager.cc:1918] T 7118043ce2504ccc854009f5293f0ace P 01ab7da76ff64100b96b1cd6f9312b8c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260621 05:11:14.875483 14345 log.cc:1196] T 7118043ce2504ccc854009f5293f0ace P 01ab7da76ff64100b96b1cd6f9312b8c: Deleting WAL directory at /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782018616648578-3598-0/minicluster-data/ts-1/wal/wals/7118043ce2504ccc854009f5293f0ace
I20260621 05:11:14.875854 14345 ts_tablet_manager.cc:1939] T 7118043ce2504ccc854009f5293f0ace P 01ab7da76ff64100b96b1cd6f9312b8c: Deleting consensus metadata
I20260621 05:11:14.876341 13827 catalog_manager.cc:4841] TS 01ab7da76ff64100b96b1cd6f9312b8c (127.3.131.130:40249): tablet 7118043ce2504ccc854009f5293f0ace (table test-table [id=9d06f49923ed4931be2930a84e266d13]) successfully deleted
W20260621 05:11:14.876466 13828 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.3.131.129:42489 (error 108)
I20260621 05:11:14.876973 14316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:11:14.876854 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.131.129:42489 (local address 127.0.0.1:54902)
0621 05:11:14.876899 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0621 05:11:14.876941 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.131.129:42489: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
W20260621 05:11:14.877085 13828 catalog_manager.cc:4584] TS 02aa1e50c2f14333bcc546ee8be8e663 (127.3.131.129:42489): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 7118043ce2504ccc854009f5293f0ace: Network error: Client connection negotiation failed: client connection to 127.3.131.129:42489: connect: Connection refused (error 111)
I20260621 05:11:14.877347 3598 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWZOx2_/build/release/bin/kudu with pid 14039
I20260621 05:11:14.883502 3598 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWZOx2_/build/release/bin/kudu with pid 14173
I20260621 05:11:14.888726 3598 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWZOx2_/build/release/bin/kudu with pid 13811
2026-06-21T05:11:14Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (2240 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782018616648578-3598-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 21 05:11:14 dist-test-slave-vrq1 krb5kdc[14349](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 21 05:11:14 dist-test-slave-vrq1 krb5kdc[14349](info): set up 2 sockets
Jun 21 05:11:14 dist-test-slave-vrq1 krb5kdc[14349](info): commencing operation
krb5kdc: starting...
W20260621 05:11:15.948397 3598 mini_kdc.cc:121] Time spent starting KDC: real 1.039s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782018616648578-3598-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782018616648578-3598-0/krb5kdc/test-user.keytab.
Jun 21 05:11:16 dist-test-slave-vrq1 krb5kdc[14349](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782018676, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-21T05:11:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-21T05:11:16Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.3.131.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.3.131.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.3.131.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782018616648578-3598-0/krb5kdc/rangeradmin_127.3.131.148@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.3.131.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782018616648578-3598-0/krb5kdc/rangeradmin_127.3.131.148@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.3.131.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.3.131.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.3.131.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782018616648578-3598-0/krb5kdc/rangerlookup_127.3.131.148@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.3.131.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782018616648578-3598-0/krb5kdc/rangerlookup_127.3.131.148@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.3.131.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.3.131.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.3.131.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782018616648578-3598-0/krb5kdc/HTTP_127.3.131.148@KRBTEST.COM.keytab.
Entry for principal HTTP/127.3.131.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782018616648578-3598-0/krb5kdc/HTTP_127.3.131.148@KRBTEST.COM.keytab.
I20260621 05:11:16.123581 3598 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782018616648578-3598-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskWZOx2_/build/release/bin/postgres/pg_ctl -D /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782018616648578-3598-0/minicluster-data/postgres -l logfile start
2026-06-21 05:11:19.015 UTC [14379] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-06-21 05:11:19.015 UTC [14379] LOG: listening on IPv4 address "127.3.131.148", port 49451
2026-06-21 05:11:19.020 UTC [14379] LOG: listening on Unix socket "/tmp/.s.PGSQL.49451"
2026-06-21 05:11:19.056 UTC [14382] LOG: database system was shut down at 2026-06-21 05:11:17 UTC
2026-06-21 05:11:19.060 UTC [14379] LOG: database system is ready to accept connections
I20260621 05:11:20.062023 3598 mini_postgres.cc:96] Postgres bound to 49451
2026-06-21 05:11:20.068 UTC [14390] FATAL: database "slave" does not exist
127.3.131.148:49451 - accepting connections
I20260621 05:11:20.069164 3598 mini_ranger.cc:161] Starting Ranger...
I20260621 05:11:20.085179 3598 mini_ranger.cc:84] Created miniranger Postgres user
I20260621 05:11:20.641840 3598 mini_ranger.cc:87] Created ranger Postgres database
I20260621 05:11:20.641928 3598 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782018616648578-3598-0/minicluster-data/ranger-admin
2026-06-21 05:11:20,939 [I] DB FLAVOR :POSTGRES
2026-06-21 05:11:20,940 [I] --------- Verifying Ranger DB connection ---------
2026-06-21 05:11:20,940 [I] Checking connection..
2026-06-21 05:11:20,940 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWZOx2_/build/release/bin/postgresql.jar:/tmp/dist-test-taskWZOx2_/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.131.148:49451/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-06-21 05:11:21,201 [I] Checking connection passed.
2026-06-21 05:11:21,201 [I] --------- Verifying version history table ---------
2026-06-21 05:11:21,201 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWZOx2_/build/release/bin/postgresql.jar:/tmp/dist-test-taskWZOx2_/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.131.148:49451/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-21 05:11:21,466 [I] Table x_db_version_h does not exist in database ranger
2026-06-21 05:11:21,467 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWZOx2_/build/release/bin/postgresql.jar:/tmp/dist-test-taskWZOx2_/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.131.148:49451/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-21 05:11:21,736 [I] Table x_db_version_h does not exist in database ranger
2026-06-21 05:11:21,736 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-06-21 05:11:21,736 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWZOx2_/build/release/bin/postgresql.jar:/tmp/dist-test-taskWZOx2_/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.131.148:49451/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskWZOx2_/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-06-21 05:11:21.990 UTC [14454] WARNING: there is no transaction in progress
2026-06-21 05:11:21,996 [I] create_dbversion_catalog.sql file imported successfully
2026-06-21 05:11:21,997 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWZOx2_/build/release/bin/postgresql.jar:/tmp/dist-test-taskWZOx2_/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.131.148:49451/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-21 05:11:22,270 [I] Table x_db_version_h already exists in database 'ranger'
2026-06-21 05:11:22,271 [I] --------- Importing Ranger Core DB Schema ---------
2026-06-21 05:11:22,271 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWZOx2_/build/release/bin/postgresql.jar:/tmp/dist-test-taskWZOx2_/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.131.148:49451/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-06-21 05:11:22,512 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWZOx2_/build/release/bin/postgresql.jar:/tmp/dist-test-taskWZOx2_/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.131.148:49451/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-06-21 05:11:22,761 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWZOx2_/build/release/bin/postgresql.jar:/tmp/dist-test-taskWZOx2_/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.131.148:49451/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-vrq1.c.gcp-upstream.internal','N') ;"
2026-06-21 05:11:23,013 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWZOx2_/build/release/bin/postgresql.jar:/tmp/dist-test-taskWZOx2_/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.131.148:49451/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-06-21 05:11:23,256 [I] Table x_portal_user does not exist in database ranger
2026-06-21 05:11:23,257 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWZOx2_/build/release/bin/postgresql.jar:/tmp/dist-test-taskWZOx2_/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.131.148:49451/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-06-21 05:11:23,508 [I] Table x_policy_ref_group does not exist in database ranger
2026-06-21 05:11:23,508 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWZOx2_/build/release/bin/postgresql.jar:/tmp/dist-test-taskWZOx2_/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.131.148:49451/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-06-21 05:11:23,750 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWZOx2_/build/release/bin/postgresql.jar:/tmp/dist-test-taskWZOx2_/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.131.148:49451/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-06-21 05:11:24,026 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-06-21 05:11:24,027 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWZOx2_/build/release/bin/postgresql.jar:/tmp/dist-test-taskWZOx2_/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.131.148:49451/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskWZOx2_/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-06-21 05:11:25.145 UTC [14562] WARNING: there is no transaction in progress
2026-06-21 05:11:25.163 UTC [14562] WARNING: there is no transaction in progress
2026-06-21 05:11:25.181 UTC [14562] WARNING: there is no transaction in progress
2026-06-21 05:11:25.199 UTC [14562] WARNING: there is no transaction in progress
2026-06-21 05:11:25.219 UTC [14562] WARNING: there is no transaction in progress
2026-06-21 05:11:25.241 UTC [14562] WARNING: there is no transaction in progress
2026-06-21 05:11:25.314 UTC [14562] WARNING: there is no transaction in progress
2026-06-21 05:11:25.321 UTC [14562] WARNING: there is no transaction in progress
2026-06-21 05:11:25.337 UTC [14562] WARNING: there is no transaction in progress
2026-06-21 05:11:25.348 UTC [14562] WARNING: there is no transaction in progress
2026-06-21 05:11:25.359 UTC [14562] WARNING: there is no transaction in progress
2026-06-21 05:11:25.376 UTC [14562] WARNING: there is no transaction in progress
2026-06-21 05:11:25.387 UTC [14562] WARNING: there is no transaction in progress
2026-06-21 05:11:25.394 UTC [14562] WARNING: there is no transaction in progress
2026-06-21 05:11:25.401 UTC [14562] WARNING: there is no transaction in progress
2026-06-21 05:11:25.409 UTC [14562] WARNING: there is no transaction in progress
2026-06-21 05:11:26,186 [I] ranger_core_db_postgres.sql file imported successfully
2026-06-21 05:11:26,186 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWZOx2_/build/release/bin/postgresql.jar:/tmp/dist-test-taskWZOx2_/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.131.148:49451/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-06-21 05:11:26,429 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-06-21 05:11:26,430 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWZOx2_/build/release/bin/postgresql.jar:/tmp/dist-test-taskWZOx2_/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.131.148:49451/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-06-21 05:11:26,696 [I] Table x_portal_user already exists in database 'ranger'
2026-06-21 05:11:26,697 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWZOx2_/build/release/bin/postgresql.jar:/tmp/dist-test-taskWZOx2_/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.131.148:49451/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-06-21 05:11:26,967 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-06-21 05:11:26,968 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWZOx2_/build/release/bin/postgresql.jar:/tmp/dist-test-taskWZOx2_/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.131.148:49451/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-06-21 05:11:27,215 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWZOx2_/build/release/bin/postgresql.jar:/tmp/dist-test-taskWZOx2_/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.131.148:49451/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-06-21 05:11:27,468 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWZOx2_/build/release/bin/postgresql.jar:/tmp/dist-test-taskWZOx2_/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.131.148:49451/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-vrq1.c.gcp-upstream.internal';"
2026-06-21 05:11:27,723 [I] CORE_DB_SCHEMA import status has been updated
2026-06-21 05:11:27,723 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWZOx2_/build/release/bin/postgresql.jar:/tmp/dist-test-taskWZOx2_/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.131.148:49451/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-06-21 05:11:27,967 [I] DB_PATCHES have already been applied
I20260621 05:11:27.973646 3598 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782018616648578-3598-0/minicluster-data/ranger-admin:/tmp/dist-test-taskWZOx2_/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskWZOx2_/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskWZOx2_/build/release/bin/postgresql.jar:/tmp/dist-test-taskWZOx2_/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260621 05:11:27.973727 3598 mini_ranger.cc:193] Using host: 127.3.131.148
I20260621 05:11:27.977470 3598 mini_ranger.cc:227] Ranger admin URL: http://127.3.131.148:43237
Jun 21, 2026 5:11:28 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jun 21, 2026 5:11:28 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jun 21, 2026 5:11:28 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782018616648578-3598-0/minicluster-data/ranger-admin/ews/webapp
Jun 21, 2026 5:11:28 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782018616648578-3598-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Jun 21, 2026 5:11:28 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782018616648578-3598-0/minicluster-data/ranger-admin/ews/webapp] .....
Jun 21, 2026 5:11:28 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782018616648578-3598-0/minicluster-data/ranger-admin/ews/webapp].
Jun 21, 2026 5:11:28 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260621 05:11:29.118477 3598 mini_ranger.cc:160] Time spent starting Ranger: real 9.049s user 0.000s sys 0.010s
Jun 21, 2026 5:11:49 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Jun 21, 2026 5:11:49 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Jun 21, 2026 5:11:49 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Jun 21, 2026 5:11:49 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Jun 21, 2026 5:11:49 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Jun 21, 2026 5:11:49 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Jun 21, 2026 5:11:49 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Jun 21, 2026 5:11:49 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Jun 21, 2026 5:11:49 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Jun 21, 2026 5:11:49 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Jun 21, 2026 5:11:49 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Jun 21, 2026 5:11:49 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Jun 21, 2026 5:11:49 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Jun 21, 2026 5:11:49 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Jun 21, 2026 5:11:49 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Jun 21, 2026 5:11:49 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Jun 21, 2026 5:11:49 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Jun 21, 2026 5:11:50 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260621 05:11:51.569214 3598 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.3.131.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.3.131.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.3.131.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782018616648578-3598-0/krb5kdc/rangerkms_127.3.131.148@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.3.131.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782018616648578-3598-0/krb5kdc/rangerkms_127.3.131.148@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.3.131.148@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.3.131.148@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.3.131.148@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782018616648578-3598-0/krb5kdc/HTTP_127.3.131.148@KRBTEST.COM.keytab.
Entry for principal HTTP/127.3.131.148@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782018616648578-3598-0/krb5kdc/HTTP_127.3.131.148@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Jun 21 05:11:51 dist-test-slave-vrq1 krb5kdc[14349](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782018711, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260621 05:11:51.663496 3598 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260621 05:11:51.675096 3598 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260621 05:11:52.797068 3598 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260621 05:11:52.797145 3598 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782018616648578-3598-0/minicluster-data/ranger-kms
I20260621 05:11:52.797158 3598 mini_ranger_kms.cc:227] Using postgres at 127.3.131.148:49451
2026-06-21 05:11:52,976 [I] DB FLAVOR :POSTGRES
2026-06-21 05:11:52,976 [I] --------- Verifying Ranger DB connection ---------
2026-06-21 05:11:52,976 [I] Checking connection
2026-06-21 05:11:52,977 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWZOx2_/build/release/bin/postgresql.jar:/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782018616648578-3598-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.131.148:49451/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-06-21 05:11:53,236 [I] connection success
2026-06-21 05:11:53,236 [I] --------- Verifying Ranger DB tables ---------
2026-06-21 05:11:53,236 [I] Verifying table ranger_masterkey in database rangerkms
2026-06-21 05:11:53,236 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWZOx2_/build/release/bin/postgresql.jar:/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782018616648578-3598-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.131.148:49451/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-06-21 05:11:53,493 [I] Table ranger_masterkey does not exist in database rangerkms
2026-06-21 05:11:53,493 [I] --------- Importing Ranger Core DB Schema ---------
2026-06-21 05:11:53,493 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-06-21 05:11:53,493 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWZOx2_/build/release/bin/postgresql.jar:/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782018616648578-3598-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.131.148:49451/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782018616648578-3598-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-06-21 05:11:53,784 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260621 05:11:54.265367 3598 mini_ranger_kms.cc:310] Created kms service
I20260621 05:11:54.483805 3598 mini_ranger_kms.cc:326] Created kudu user
I20260621 05:11:54.529130 3598 mini_ranger_kms.cc:343] Created rangerkms user
Jun 21 05:11:54 dist-test-slave-vrq1 krb5kdc[14349](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782018711, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.3.131.148@KRBTEST.COM
I20260621 05:11:55.014796 3598 mini_ranger_kms.cc:384] Added ranger policy
I20260621 05:11:55.014919 3598 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782018616648578-3598-0/minicluster-data/ranger-kms:/tmp/dist-test-taskWZOx2_/build/release/bin/postgresql.jar:/tmp/dist-test-taskWZOx2_/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskWZOx2_/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260621 05:11:55.014941 3598 mini_ranger_kms.cc:240] Using host: 127.3.131.148
I20260621 05:11:55.018421 3598 mini_ranger_kms.cc:276] Ranger KMS PID: 14743
I20260621 05:11:55.018560 3598 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.3.131.148:46463
Jun 21, 2026 5:11:56 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jun 21, 2026 5:11:56 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jun 21, 2026 5:11:56 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782018616648578-3598-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Jun 21, 2026 5:11:56 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782018616648578-3598-0/minicluster-data/ranger-kms/ews/webapp] .....
Jun 21, 2026 5:11:56 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782018616648578-3598-0/minicluster-data/ranger-kms/ews/webapp].
Jun 21, 2026 5:11:56 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260621 05:11:57.376217 3598 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.713s user 0.000s sys 0.162s
I20260621 05:11:57.376329 3598 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260621 05:11:57.376402 3598 mini_ranger_kms.cc:401] 127.3.131.148:46463/kms/v1/keys
Jun 21 05:11:57 dist-test-slave-vrq1 krb5kdc[14349](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1782018717, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.3.131.148@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Jun 21 05:11:57 dist-test-slave-vrq1 krb5kdc[14349](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782018717, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.3.131.148@KRBTEST.COM for HTTP/127.3.131.148@KRBTEST.COM
Jun 21 05:11:58 dist-test-slave-vrq1 krb5kdc[14349](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782018717, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.3.131.148@KRBTEST.COM for HTTP/127.3.131.148@KRBTEST.COM
Jun 21 05:12:00 dist-test-slave-vrq1 krb5kdc[14349](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782018720, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.3.131.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.131.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.131.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782018616648578-3598-0/krb5kdc/kudu_127.3.131.190.keytab.
Entry for principal kudu/127.3.131.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782018616648578-3598-0/krb5kdc/kudu_127.3.131.190.keytab.
I20260621 05:12:00.739877 3598 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWZOx2_/build/release/bin/kudu
/tmp/dist-test-taskWZOx2_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782018616648578-3598-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782018616648578-3598-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782018616648578-3598-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782018616648578-3598-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.131.190:35685
--webserver_interface=127.3.131.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782018616648578-3598-0/krb5kdc/kudu_127.3.131.190.keytab
--principal=kudu/127.3.131.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.131.148:41299
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.131.190:35685
--ranger_config_path=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782018616648578-3598-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.3.131.148:46463/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782018616648578-3598-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782018616648578-3598-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782018616648578-3598-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 05:12:00.833792 14793 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 05:12:00.834096 14793 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 05:12:00.834121 14793 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 05:12:00.835475 14793 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 05:12:00.835531 14793 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 05:12:00.835565 14793 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 05:12:00.835578 14793 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 05:12:00.835590 14793 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 05:12:00.837512 14793 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.131.148:41299
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782018616648578-3598-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782018616648578-3598-0/minicluster-data/master-0/wal
--ranger_kms_url=127.3.131.148:46463/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.3.131.190:35685
--ranger_config_path=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782018616648578-3598-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782018616648578-3598-0/krb5kdc/kudu_127.3.131.190.keytab
--principal=kudu/127.3.131.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.131.190:35685
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782018616648578-3598-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.131.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14793
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782018616648578-3598-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 21 Jun 2026 04:16:16 UTC on ccycloud.quasar-xycaha.root.comops.site
build id 4457881
I20260621 05:12:00.837894 14793 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 05:12:00.838197 14793 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260621 05:12:00.845224 14793 server_base.cc:1034] running on GCE node
W20260621 05:12:00.845224 14799 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:12:00.845322 14801 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:12:00.846217 14798 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:12:00.846580 14793 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 05:12:00.847147 14793 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 05:12:00.848397 14793 hybrid_clock.cc:648] HybridClock initialized: now 1782018720848382 us; error 43 us; skew 500 ppm
Jun 21 05:12:00 dist-test-slave-vrq1 krb5kdc[14349](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782018720, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.131.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 05:12:00.857448 14793 init.cc:377] Logged in from keytab as kudu/127.3.131.190@KRBTEST.COM (short username kudu)
I20260621 05:12:00.858065 14793 webserver.cc:460] Webserver started at http://127.3.131.190:36067/ using document root <none> and password file <none>
I20260621 05:12:00.858325 14793 fs_manager.cc:359] Metadata directory not provided
I20260621 05:12:00.858506 14793 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 05:12:00.858680 14793 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 21 05:12:00 dist-test-slave-vrq1 krb5kdc[14349](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782018720, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.131.190@KRBTEST.COM for HTTP/127.3.131.148@KRBTEST.COM
I20260621 05:12:00.914917 14793 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782018616648578-3598-0/minicluster-data/master-0/data/instance:
uuid: "8c591b6c189f45efad224430a2e55fcb"
format_stamp: "Formatted at 2026-06-21 05:12:00 on dist-test-slave-vrq1"
server_key: "ab6496b393c8b36c4ab043a3fd7a2903"
server_key_iv: "a2b1b9be5e0b09a3936862d54e8603a7"
server_key_version: "kuduclusterkey@0"
I20260621 05:12:00.915452 14793 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782018616648578-3598-0/minicluster-data/master-0/wal/instance:
uuid: "8c591b6c189f45efad224430a2e55fcb"
format_stamp: "Formatted at 2026-06-21 05:12:00 on dist-test-slave-vrq1"
server_key: "ab6496b393c8b36c4ab043a3fd7a2903"
server_key_iv: "a2b1b9be5e0b09a3936862d54e8603a7"
server_key_version: "kuduclusterkey@0"
I20260621 05:12:00.917212 14793 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.002s sys 0.000s
I20260621 05:12:00.930449 14808 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 05:12:00.931015 14793 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260621 05:12:00.931151 14793 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782018616648578-3598-0/minicluster-data/master-0/data,/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782018616648578-3598-0/minicluster-data/master-0/wal
uuid: "8c591b6c189f45efad224430a2e55fcb"
format_stamp: "Formatted at 2026-06-21 05:12:00 on dist-test-slave-vrq1"
server_key: "ab6496b393c8b36c4ab043a3fd7a2903"
server_key_iv: "a2b1b9be5e0b09a3936862d54e8603a7"
server_key_version: "kuduclusterkey@0"
I20260621 05:12:00.931284 14793 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782018616648578-3598-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782018616648578-3598-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782018616648578-3598-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 05:12:00.956178 14793 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 05:12:00.958336 14793 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 05:12:00.958515 14793 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 05:12:00.963822 14793 rpc_server.cc:307] RPC server started. Bound to: 127.3.131.190:35685
I20260621 05:12:00.963879 14860 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.131.190:35685 every 8 connection(s)
I20260621 05:12:00.964270 14793 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782018616648578-3598-0/minicluster-data/master-0/data/info.pb
I20260621 05:12:00.964963 14861 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 05:12:00.967211 14861 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8c591b6c189f45efad224430a2e55fcb: Bootstrap starting.
I20260621 05:12:00.967746 14861 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8c591b6c189f45efad224430a2e55fcb: Neither blocks nor log segments found. Creating new log.
I20260621 05:12:00.968122 3598 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWZOx2_/build/release/bin/kudu as pid 14793
I20260621 05:12:00.968186 14861 log.cc:824] T 00000000000000000000000000000000 P 8c591b6c189f45efad224430a2e55fcb: Log is configured to *not* fsync() on all Append() calls
I20260621 05:12:00.968231 3598 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782018616648578-3598-0/minicluster-data/master-0/wal/instance
I20260621 05:12:00.968441 3598 external_mini_cluster.cc:1428] Setting key 814ebc99b9e29946609a6989d7500329
I20260621 05:12:00.969003 14861 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8c591b6c189f45efad224430a2e55fcb: No bootstrap required, opened a new log
I20260621 05:12:00.971674 14861 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8c591b6c189f45efad224430a2e55fcb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c591b6c189f45efad224430a2e55fcb" member_type: VOTER last_known_addr { host: "127.3.131.190" port: 35685 } }
I20260621 05:12:00.971848 14861 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8c591b6c189f45efad224430a2e55fcb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 05:12:00.971909 14861 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8c591b6c189f45efad224430a2e55fcb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8c591b6c189f45efad224430a2e55fcb, State: Initialized, Role: FOLLOWER
I20260621 05:12:00.972061 14861 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8c591b6c189f45efad224430a2e55fcb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c591b6c189f45efad224430a2e55fcb" member_type: VOTER last_known_addr { host: "127.3.131.190" port: 35685 } }
I20260621 05:12:00.972164 14861 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8c591b6c189f45efad224430a2e55fcb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260621 05:12:00.972210 14861 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8c591b6c189f45efad224430a2e55fcb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260621 05:12:00.972265 14861 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8c591b6c189f45efad224430a2e55fcb [term 0 FOLLOWER]: Advancing to term 1
I20260621 05:12:00.972842 14861 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8c591b6c189f45efad224430a2e55fcb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c591b6c189f45efad224430a2e55fcb" member_type: VOTER last_known_addr { host: "127.3.131.190" port: 35685 } }
I20260621 05:12:00.972963 14861 leader_election.cc:302] T 00000000000000000000000000000000 P 8c591b6c189f45efad224430a2e55fcb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8c591b6c189f45efad224430a2e55fcb; no voters:
I20260621 05:12:00.973146 14861 leader_election.cc:288] T 00000000000000000000000000000000 P 8c591b6c189f45efad224430a2e55fcb [CANDIDATE]: Term 1 election: Requested vote from peers
I20260621 05:12:00.973198 14866 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8c591b6c189f45efad224430a2e55fcb [term 1 FOLLOWER]: Leader election won for term 1
I20260621 05:12:00.973394 14861 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8c591b6c189f45efad224430a2e55fcb [sys.catalog]: configured and running, proceeding with master startup.
I20260621 05:12:00.973392 14866 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8c591b6c189f45efad224430a2e55fcb [term 1 LEADER]: Becoming Leader. State: Replica: 8c591b6c189f45efad224430a2e55fcb, State: Running, Role: LEADER
I20260621 05:12:00.973634 14866 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8c591b6c189f45efad224430a2e55fcb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c591b6c189f45efad224430a2e55fcb" member_type: VOTER last_known_addr { host: "127.3.131.190" port: 35685 } }
I20260621 05:12:00.974045 14868 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8c591b6c189f45efad224430a2e55fcb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8c591b6c189f45efad224430a2e55fcb. Latest consensus state: current_term: 1 leader_uuid: "8c591b6c189f45efad224430a2e55fcb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c591b6c189f45efad224430a2e55fcb" member_type: VOTER last_known_addr { host: "127.3.131.190" port: 35685 } } }
I20260621 05:12:00.974212 14868 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8c591b6c189f45efad224430a2e55fcb [sys.catalog]: This master's current role is: LEADER
I20260621 05:12:00.974423 14867 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8c591b6c189f45efad224430a2e55fcb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8c591b6c189f45efad224430a2e55fcb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c591b6c189f45efad224430a2e55fcb" member_type: VOTER last_known_addr { host: "127.3.131.190" port: 35685 } } }
I20260621 05:12:00.974561 14867 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8c591b6c189f45efad224430a2e55fcb [sys.catalog]: This master's current role is: LEADER
I20260621 05:12:00.974488 14875 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260621 05:12:00.975023 14861 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782018616648578-3598-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260621 05:12:00.975302 14875 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Jun 21 05:12:00 dist-test-slave-vrq1 krb5kdc[14349](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782018720, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.131.190@KRBTEST.COM
I20260621 05:12:00.981360 14875 catalog_manager.cc:1347] Generated new cluster ID: 72238250015b40d0bbd65360233c3b99
I20260621 05:12:00.981513 14875 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260621 05:12:00.985709 14865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:12:00.968997 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45462 (local address 127.3.131.190:35685)
0621 05:12:00.969172 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0621 05:12:00.969182 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0621 05:12:00.969217 (+ 35us) server_negotiation.cc:407] Connection header received
0621 05:12:00.971680 (+ 2463us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:12:00.971690 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:12:00.971764 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:12:00.971983 (+ 219us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:12:00.972665 (+ 682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:12:00.973999 (+ 1334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:12:00.975036 (+ 1037us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:12:00.977884 (+ 2848us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:12:00.978207 (+ 323us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:12:00.981473 (+ 3266us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:12:00.981478 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:12:00.981478 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:12:00.981493 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:12:00.983901 (+ 2408us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:12:00.984382 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:12:00.984385 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:12:00.984385 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:12:00.984498 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:12:00.984715 (+ 217us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:12:00.984715 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:12:00.984716 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:12:00.984993 (+ 277us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:12:00.985157 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0621 05:12:00.985269 (+ 112us) server_negotiation.cc:299] Negotiation successful
0621 05:12:00.985410 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":122,"thread_start_us":92,"threads_started":1}
I20260621 05:12:01.038105 14875 catalog_manager.cc:1370] Generated new certificate authority record
I20260621 05:12:01.038909 14875 catalog_manager.cc:1504] Loading token signing keys...
I20260621 05:12:01.045311 14875 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8c591b6c189f45efad224430a2e55fcb: Generated new TSK 0
I20260621 05:12:01.045568 14875 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.3.131.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.131.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.131.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782018616648578-3598-0/krb5kdc/kudu_127.3.131.129.keytab.
Entry for principal kudu/127.3.131.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782018616648578-3598-0/krb5kdc/kudu_127.3.131.129.keytab.
I20260621 05:12:01.765661 3598 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWZOx2_/build/release/bin/kudu
/tmp/dist-test-taskWZOx2_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782018616648578-3598-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782018616648578-3598-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782018616648578-3598-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782018616648578-3598-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.3.131.148:46463/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.131.129:0
--local_ip_for_outbound_sockets=127.3.131.129
--webserver_interface=127.3.131.129
--webserver_port=0
--tserver_master_addrs=127.3.131.190:35685
--keytab_file=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782018616648578-3598-0/krb5kdc/kudu_127.3.131.129.keytab
--principal=kudu/127.3.131.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.131.148:41299
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782018616648578-3598-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782018616648578-3598-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782018616648578-3598-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 05:12:01.904318 14906 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 05:12:01.904775 14906 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 05:12:01.904827 14906 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 05:12:01.907539 14906 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 05:12:01.907624 14906 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 05:12:01.907820 14906 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.131.129
I20260621 05:12:01.911240 14906 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.131.148:41299
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782018616648578-3598-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782018616648578-3598-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.3.131.148:46463/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782018616648578-3598-0/krb5kdc/kudu_127.3.131.129.keytab
--principal=kudu/127.3.131.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.131.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782018616648578-3598-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.131.129
--webserver_port=0
--tserver_master_addrs=127.3.131.190:35685
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14906
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.131.129
--log_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782018616648578-3598-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 21 Jun 2026 04:16:16 UTC on ccycloud.quasar-xycaha.root.comops.site
build id 4457881
I20260621 05:12:01.911574 14906 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 05:12:01.911867 14906 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260621 05:12:01.921160 14906 server_base.cc:1034] running on GCE node
W20260621 05:12:01.921195 14911 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:12:01.921363 14914 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:12:01.921401 14912 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:12:01.921833 14906 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 05:12:01.922207 14906 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 05:12:01.923417 14906 hybrid_clock.cc:648] HybridClock initialized: now 1782018721923397 us; error 39 us; skew 500 ppm
Jun 21 05:12:01 dist-test-slave-vrq1 krb5kdc[14349](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782018721, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.131.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 05:12:01.928175 14906 init.cc:377] Logged in from keytab as kudu/127.3.131.129@KRBTEST.COM (short username kudu)
I20260621 05:12:01.928951 14906 webserver.cc:460] Webserver started at http://127.3.131.129:42165/ using document root <none> and password file <none>
I20260621 05:12:01.929791 14906 fs_manager.cc:359] Metadata directory not provided
I20260621 05:12:01.929899 14906 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 05:12:01.930058 14906 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 21 05:12:01 dist-test-slave-vrq1 krb5kdc[14349](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782018721, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.131.129@KRBTEST.COM for HTTP/127.3.131.148@KRBTEST.COM
I20260621 05:12:01.945663 14906 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782018616648578-3598-0/minicluster-data/ts-0/data/instance:
uuid: "c51448aadc654a55bd7162d5b990579f"
format_stamp: "Formatted at 2026-06-21 05:12:01 on dist-test-slave-vrq1"
server_key: "59350048ff288db0eee76da58e478d7d"
server_key_iv: "95db7eaf59e90d15a378bf2c6313f443"
server_key_version: "kuduclusterkey@0"
I20260621 05:12:01.946138 14906 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782018616648578-3598-0/minicluster-data/ts-0/wal/instance:
uuid: "c51448aadc654a55bd7162d5b990579f"
format_stamp: "Formatted at 2026-06-21 05:12:01 on dist-test-slave-vrq1"
server_key: "59350048ff288db0eee76da58e478d7d"
server_key_iv: "95db7eaf59e90d15a378bf2c6313f443"
server_key_version: "kuduclusterkey@0"
I20260621 05:12:01.951012 14906 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.000s
I20260621 05:12:01.970791 14921 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 05:12:01.971050 14906 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.002s
I20260621 05:12:01.971144 14906 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782018616648578-3598-0/minicluster-data/ts-0/data,/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782018616648578-3598-0/minicluster-data/ts-0/wal
uuid: "c51448aadc654a55bd7162d5b990579f"
format_stamp: "Formatted at 2026-06-21 05:12:01 on dist-test-slave-vrq1"
server_key: "59350048ff288db0eee76da58e478d7d"
server_key_iv: "95db7eaf59e90d15a378bf2c6313f443"
server_key_version: "kuduclusterkey@0"
I20260621 05:12:01.971292 14906 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782018616648578-3598-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782018616648578-3598-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782018616648578-3598-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 05:12:01.993634 14906 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 05:12:01.996358 14906 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 05:12:01.996529 14906 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 05:12:01.996826 14906 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 05:12:01.997166 14906 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 05:12:01.997224 14906 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 05:12:01.997267 14906 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 05:12:01.997295 14906 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 05:12:02.007387 14906 rpc_server.cc:307] RPC server started. Bound to: 127.3.131.129:39409
I20260621 05:12:02.007474 15036 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.131.129:39409 every 8 connection(s)
I20260621 05:12:02.007988 14906 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782018616648578-3598-0/minicluster-data/ts-0/data/info.pb
I20260621 05:12:02.016808 3598 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWZOx2_/build/release/bin/kudu as pid 14906
I20260621 05:12:02.016942 3598 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782018616648578-3598-0/minicluster-data/ts-0/wal/instance
I20260621 05:12:02.017163 3598 external_mini_cluster.cc:1428] Setting key 731f2a62d502a79ac4cd478fa46da757
Jun 21 05:12:02 dist-test-slave-vrq1 krb5kdc[14349](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782018721, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.131.129@KRBTEST.COM for kudu/127.3.131.190@KRBTEST.COM
I20260621 05:12:02.022117 15040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:12:02.009132 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.131.129:54531 (local address 127.3.131.190:35685)
0621 05:12:02.009274 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0621 05:12:02.009277 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0621 05:12:02.009588 (+ 311us) server_negotiation.cc:407] Connection header received
0621 05:12:02.010243 (+ 655us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:12:02.010245 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:12:02.010260 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:12:02.010351 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:12:02.011390 (+ 1039us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:12:02.012218 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:12:02.014116 (+ 1898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:12:02.014423 (+ 307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:12:02.014509 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:12:02.017931 (+ 3422us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:12:02.017933 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:12:02.017933 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:12:02.017942 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:12:02.019865 (+ 1923us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:12:02.020490 (+ 625us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:12:02.020492 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:12:02.020492 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:12:02.020586 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:12:02.020991 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:12:02.020993 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:12:02.020993 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:12:02.021165 (+ 172us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:12:02.021257 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0621 05:12:02.021896 (+ 639us) server_negotiation.cc:299] Negotiation successful
0621 05:12:02.021987 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":115,"thread_start_us":82,"threads_started":1}
I20260621 05:12:02.022258 15039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:12:02.008999 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.131.190:35685 (local address 127.3.131.129:54531)
0621 05:12:02.009294 (+ 295us) negotiation.cc:107] Waiting for socket to connect
0621 05:12:02.009344 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0621 05:12:02.010103 (+ 759us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:12:02.010459 (+ 356us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:12:02.010464 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:12:02.010794 (+ 330us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:12:02.011164 (+ 370us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:12:02.011253 (+ 89us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:12:02.012348 (+ 1095us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:12:02.012350 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:12:02.013994 (+ 1644us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:12:02.013998 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:12:02.014534 (+ 536us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:12:02.014536 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:12:02.014700 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:12:02.015643 (+ 943us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 05:12:02.015660 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:12:02.017765 (+ 2105us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 05:12:02.019998 (+ 2233us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:12:02.020010 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:12:02.020014 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:12:02.020370 (+ 356us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:12:02.020675 (+ 305us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:12:02.020676 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:12:02.020677 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:12:02.020847 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:12:02.021274 (+ 427us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 05:12:02.021283 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 05:12:02.021488 (+ 205us) client_negotiation.cc:769] Sending connection context
0621 05:12:02.021626 (+ 138us) client_negotiation.cc:240] Negotiation successful
0621 05:12:02.021973 (+ 347us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":249,"thread_start_us":92,"threads_started":1}
I20260621 05:12:02.022955 15037 heartbeater.cc:347] Connected to a master server at 127.3.131.190:35685
I20260621 05:12:02.023064 15037 heartbeater.cc:464] Registering TS with master...
I20260621 05:12:02.023296 15037 heartbeater.cc:511] Master 127.3.131.190:35685 requested a full tablet report, sending...
I20260621 05:12:02.023838 14825 ts_manager.cc:194] Registered new tserver with Master: c51448aadc654a55bd7162d5b990579f (127.3.131.129:39409)
I20260621 05:12:02.024793 14825 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.131.129@KRBTEST.COM'} at 127.3.131.129:54531
WARNING: no policy specified for kudu/127.3.131.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.131.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.131.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782018616648578-3598-0/krb5kdc/kudu_127.3.131.130.keytab.
Entry for principal kudu/127.3.131.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782018616648578-3598-0/krb5kdc/kudu_127.3.131.130.keytab.
I20260621 05:12:02.067477 3598 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWZOx2_/build/release/bin/kudu
/tmp/dist-test-taskWZOx2_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782018616648578-3598-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782018616648578-3598-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782018616648578-3598-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782018616648578-3598-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.3.131.148:46463/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.131.130:0
--local_ip_for_outbound_sockets=127.3.131.130
--webserver_interface=127.3.131.130
--webserver_port=0
--tserver_master_addrs=127.3.131.190:35685
--keytab_file=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782018616648578-3598-0/krb5kdc/kudu_127.3.131.130.keytab
--principal=kudu/127.3.131.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.131.148:41299
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782018616648578-3598-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782018616648578-3598-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782018616648578-3598-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 05:12:02.209894 15043 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 05:12:02.210142 15043 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 05:12:02.210187 15043 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 05:12:02.211690 15043 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 05:12:02.211764 15043 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 05:12:02.211815 15043 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.131.130
Jun 21 05:12:02 dist-test-slave-vrq1 krb5kdc[14349](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1782018722, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.131.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 05:12:02.213696 15043 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.131.148:41299
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782018616648578-3598-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782018616648578-3598-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.3.131.148:46463/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782018616648578-3598-0/krb5kdc/kudu_127.3.131.130.keytab
--principal=kudu/127.3.131.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.131.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782018616648578-3598-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.131.130
--webserver_port=0
--tserver_master_addrs=127.3.131.190:35685
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15043
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.131.130
--log_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782018616648578-3598-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 21 Jun 2026 04:16:16 UTC on ccycloud.quasar-xycaha.root.comops.site
build id 4457881
I20260621 05:12:02.213982 15043 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 05:12:02.214257 15043 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 05:12:02.220657 15051 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:12:02.221534 15053 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:12:02.221874 15050 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:12:02.222493 15043 server_base.cc:1034] running on GCE node
I20260621 05:12:02.222725 15043 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 05:12:02.223043 15043 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 05:12:02.230512 15043 hybrid_clock.cc:648] HybridClock initialized: now 1782018722230467 us; error 50 us; skew 500 ppm
Jun 21 05:12:02 dist-test-slave-vrq1 krb5kdc[14349](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782018722, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.131.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 05:12:02.234889 15043 init.cc:377] Logged in from keytab as kudu/127.3.131.130@KRBTEST.COM (short username kudu)
I20260621 05:12:02.235677 15043 webserver.cc:460] Webserver started at http://127.3.131.130:37663/ using document root <none> and password file <none>
I20260621 05:12:02.236056 15043 fs_manager.cc:359] Metadata directory not provided
I20260621 05:12:02.236140 15043 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 05:12:02.236299 15043 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 21 05:12:02 dist-test-slave-vrq1 krb5kdc[14349](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782018722, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.131.130@KRBTEST.COM for HTTP/127.3.131.148@KRBTEST.COM
I20260621 05:12:02.255656 15043 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782018616648578-3598-0/minicluster-data/ts-1/data/instance:
uuid: "3e5d83cff1c440d6929fa7475977853f"
format_stamp: "Formatted at 2026-06-21 05:12:02 on dist-test-slave-vrq1"
server_key: "c391b90e750f73cb32144d9427567550"
server_key_iv: "c71913034987e673f891a3fd9dba9687"
server_key_version: "kuduclusterkey@0"
I20260621 05:12:02.256086 15043 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782018616648578-3598-0/minicluster-data/ts-1/wal/instance:
uuid: "3e5d83cff1c440d6929fa7475977853f"
format_stamp: "Formatted at 2026-06-21 05:12:02 on dist-test-slave-vrq1"
server_key: "c391b90e750f73cb32144d9427567550"
server_key_iv: "c71913034987e673f891a3fd9dba9687"
server_key_version: "kuduclusterkey@0"
I20260621 05:12:02.257860 15043 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.002s sys 0.000s
I20260621 05:12:02.272892 15060 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 05:12:02.273173 15043 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260621 05:12:02.273269 15043 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782018616648578-3598-0/minicluster-data/ts-1/data,/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782018616648578-3598-0/minicluster-data/ts-1/wal
uuid: "3e5d83cff1c440d6929fa7475977853f"
format_stamp: "Formatted at 2026-06-21 05:12:02 on dist-test-slave-vrq1"
server_key: "c391b90e750f73cb32144d9427567550"
server_key_iv: "c71913034987e673f891a3fd9dba9687"
server_key_version: "kuduclusterkey@0"
I20260621 05:12:02.273368 15043 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782018616648578-3598-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782018616648578-3598-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782018616648578-3598-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 05:12:02.311084 15043 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 05:12:02.314473 15043 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 05:12:02.314680 15043 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 05:12:02.315002 15043 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 05:12:02.315655 15043 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 05:12:02.315718 15043 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 05:12:02.315786 15043 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 05:12:02.315835 15043 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 05:12:02.326154 15043 rpc_server.cc:307] RPC server started. Bound to: 127.3.131.130:46525
I20260621 05:12:02.326596 15043 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782018616648578-3598-0/minicluster-data/ts-1/data/info.pb
I20260621 05:12:02.327342 15174 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.131.130:46525 every 8 connection(s)
Jun 21 05:12:02 dist-test-slave-vrq1 krb5kdc[14349](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782018722, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.131.130@KRBTEST.COM for kudu/127.3.131.190@KRBTEST.COM
I20260621 05:12:02.336231 3598 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWZOx2_/build/release/bin/kudu as pid 15043
I20260621 05:12:02.336355 3598 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782018616648578-3598-0/minicluster-data/ts-1/wal/instance
I20260621 05:12:02.336575 3598 external_mini_cluster.cc:1428] Setting key e9bb93245f2559e1183e67be0d7c5f7a
I20260621 05:12:02.340207 15040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:12:02.327813 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.131.130:33073 (local address 127.3.131.190:35685)
0621 05:12:02.327877 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0621 05:12:02.327880 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0621 05:12:02.328030 (+ 150us) server_negotiation.cc:407] Connection header received
0621 05:12:02.328641 (+ 611us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:12:02.328642 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:12:02.328661 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:12:02.328724 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:12:02.329638 (+ 914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:12:02.330389 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:12:02.331931 (+ 1542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:12:02.332265 (+ 334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:12:02.332357 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:12:02.336067 (+ 3710us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:12:02.336069 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:12:02.336070 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:12:02.336079 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:12:02.338216 (+ 2137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:12:02.338832 (+ 616us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:12:02.338834 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:12:02.338834 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:12:02.338910 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:12:02.339284 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:12:02.339286 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:12:02.339286 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:12:02.339460 (+ 174us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:12:02.339510 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0621 05:12:02.339990 (+ 480us) server_negotiation.cc:299] Negotiation successful
0621 05:12:02.340099 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260621 05:12:02.341548 15177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:12:02.327479 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.131.190:35685 (local address 127.3.131.130:33073)
0621 05:12:02.327858 (+ 379us) negotiation.cc:107] Waiting for socket to connect
0621 05:12:02.327908 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0621 05:12:02.328509 (+ 601us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:12:02.328951 (+ 442us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:12:02.328956 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:12:02.329172 (+ 216us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:12:02.329410 (+ 238us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:12:02.329415 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:12:02.330525 (+ 1110us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:12:02.330526 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:12:02.331814 (+ 1288us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:12:02.331818 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:12:02.332450 (+ 632us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:12:02.332453 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:12:02.332665 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:12:02.333848 (+ 1183us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 05:12:02.333860 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:12:02.335930 (+ 2070us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 05:12:02.338337 (+ 2407us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:12:02.338348 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:12:02.338351 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:12:02.338740 (+ 389us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:12:02.338993 (+ 253us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:12:02.338993 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:12:02.338994 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:12:02.339154 (+ 160us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:12:02.339565 (+ 411us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 05:12:02.339572 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 05:12:02.339807 (+ 235us) client_negotiation.cc:769] Sending connection context
0621 05:12:02.341019 (+ 1212us) client_negotiation.cc:240] Negotiation successful
0621 05:12:02.341254 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":329,"thread_start_us":102,"threads_started":1}
I20260621 05:12:02.343910 15175 heartbeater.cc:347] Connected to a master server at 127.3.131.190:35685
I20260621 05:12:02.344062 15175 heartbeater.cc:464] Registering TS with master...
I20260621 05:12:02.344306 15175 heartbeater.cc:511] Master 127.3.131.190:35685 requested a full tablet report, sending...
I20260621 05:12:02.345047 14825 ts_manager.cc:194] Registered new tserver with Master: 3e5d83cff1c440d6929fa7475977853f (127.3.131.130:46525)
I20260621 05:12:02.345681 14825 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.131.130@KRBTEST.COM'} at 127.3.131.130:33073
WARNING: no policy specified for kudu/127.3.131.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.131.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.131.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782018616648578-3598-0/krb5kdc/kudu_127.3.131.131.keytab.
Entry for principal kudu/127.3.131.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782018616648578-3598-0/krb5kdc/kudu_127.3.131.131.keytab.
I20260621 05:12:02.376816 3598 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWZOx2_/build/release/bin/kudu
/tmp/dist-test-taskWZOx2_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782018616648578-3598-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782018616648578-3598-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782018616648578-3598-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782018616648578-3598-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.3.131.148:46463/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.131.131:0
--local_ip_for_outbound_sockets=127.3.131.131
--webserver_interface=127.3.131.131
--webserver_port=0
--tserver_master_addrs=127.3.131.190:35685
--keytab_file=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782018616648578-3598-0/krb5kdc/kudu_127.3.131.131.keytab
--principal=kudu/127.3.131.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.131.148:41299
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782018616648578-3598-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782018616648578-3598-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782018616648578-3598-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 05:12:02.527211 15180 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 05:12:02.527458 15180 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 05:12:02.527491 15180 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 05:12:02.529265 15180 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 05:12:02.529345 15180 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 05:12:02.529386 15180 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.131.131
I20260621 05:12:02.531738 15180 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.131.148:41299
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782018616648578-3598-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782018616648578-3598-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.3.131.148:46463/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782018616648578-3598-0/krb5kdc/kudu_127.3.131.131.keytab
--principal=kudu/127.3.131.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.131.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782018616648578-3598-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.131.131
--webserver_port=0
--tserver_master_addrs=127.3.131.190:35685
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15180
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.131.131
--log_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782018616648578-3598-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 21 Jun 2026 04:16:16 UTC on ccycloud.quasar-xycaha.root.comops.site
build id 4457881
I20260621 05:12:02.532049 15180 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 05:12:02.532341 15180 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 05:12:02.540103 15185 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:12:02.540165 15186 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:12:02.540149 15188 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:12:02.540485 15180 server_base.cc:1034] running on GCE node
I20260621 05:12:02.540704 15180 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 05:12:02.541169 15180 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 05:12:02.542372 15180 hybrid_clock.cc:648] HybridClock initialized: now 1782018722542348 us; error 42 us; skew 500 ppm
Jun 21 05:12:02 dist-test-slave-vrq1 krb5kdc[14349](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782018722, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.131.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 05:12:02.546015 15180 init.cc:377] Logged in from keytab as kudu/127.3.131.131@KRBTEST.COM (short username kudu)
I20260621 05:12:02.546518 15180 webserver.cc:460] Webserver started at http://127.3.131.131:32879/ using document root <none> and password file <none>
I20260621 05:12:02.546803 15180 fs_manager.cc:359] Metadata directory not provided
I20260621 05:12:02.546878 15180 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 05:12:02.547041 15180 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 21 05:12:02 dist-test-slave-vrq1 krb5kdc[14349](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782018722, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.131.131@KRBTEST.COM for HTTP/127.3.131.148@KRBTEST.COM
I20260621 05:12:02.566282 15180 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782018616648578-3598-0/minicluster-data/ts-2/data/instance:
uuid: "78d6b458616b46e28882ddfc019c3dee"
format_stamp: "Formatted at 2026-06-21 05:12:02 on dist-test-slave-vrq1"
server_key: "579966ebbfee17dfb5b48e892bd3b42e"
server_key_iv: "8d095c9c4847f0aef5f3baee706a7c57"
server_key_version: "kuduclusterkey@0"
I20260621 05:12:02.575335 15180 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782018616648578-3598-0/minicluster-data/ts-2/wal/instance:
uuid: "78d6b458616b46e28882ddfc019c3dee"
format_stamp: "Formatted at 2026-06-21 05:12:02 on dist-test-slave-vrq1"
server_key: "579966ebbfee17dfb5b48e892bd3b42e"
server_key_iv: "8d095c9c4847f0aef5f3baee706a7c57"
server_key_version: "kuduclusterkey@0"
I20260621 05:12:02.577888 15180 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.002s sys 0.000s
I20260621 05:12:02.605356 15195 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 05:12:02.605693 15180 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.001s
I20260621 05:12:02.605825 15180 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782018616648578-3598-0/minicluster-data/ts-2/data,/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782018616648578-3598-0/minicluster-data/ts-2/wal
uuid: "78d6b458616b46e28882ddfc019c3dee"
format_stamp: "Formatted at 2026-06-21 05:12:02 on dist-test-slave-vrq1"
server_key: "579966ebbfee17dfb5b48e892bd3b42e"
server_key_iv: "8d095c9c4847f0aef5f3baee706a7c57"
server_key_version: "kuduclusterkey@0"
I20260621 05:12:02.605937 15180 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782018616648578-3598-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782018616648578-3598-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782018616648578-3598-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 05:12:02.632136 15180 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 05:12:02.635099 15180 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 05:12:02.635515 15180 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 05:12:02.635973 15180 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 05:12:02.636574 15180 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 05:12:02.636675 15180 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 05:12:02.636757 15180 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 05:12:02.636816 15180 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 05:12:02.646572 15180 rpc_server.cc:307] RPC server started. Bound to: 127.3.131.131:33053
I20260621 05:12:02.647130 15180 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782018616648578-3598-0/minicluster-data/ts-2/data/info.pb
I20260621 05:12:02.655260 15308 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.131.131:33053 every 8 connection(s)
Jun 21 05:12:02 dist-test-slave-vrq1 krb5kdc[14349](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782018722, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.131.131@KRBTEST.COM for kudu/127.3.131.190@KRBTEST.COM
I20260621 05:12:02.656368 3598 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWZOx2_/build/release/bin/kudu as pid 15180
I20260621 05:12:02.656486 3598 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782018616648578-3598-0/minicluster-data/ts-2/wal/instance
I20260621 05:12:02.656723 3598 external_mini_cluster.cc:1428] Setting key 7db34cc195c43df59f9ea4a301f99e04
I20260621 05:12:02.661556 15040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:12:02.648246 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.131.131:41531 (local address 127.3.131.190:35685)
0621 05:12:02.648298 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0621 05:12:02.648300 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:12:02.648579 (+ 279us) server_negotiation.cc:407] Connection header received
0621 05:12:02.649319 (+ 740us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:12:02.649320 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:12:02.649334 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:12:02.649367 (+ 33us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:12:02.650292 (+ 925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:12:02.651080 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:12:02.652695 (+ 1615us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:12:02.652995 (+ 300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:12:02.653061 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:12:02.657471 (+ 4410us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:12:02.657473 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:12:02.657473 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:12:02.657510 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:12:02.659557 (+ 2047us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:12:02.660186 (+ 629us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:12:02.660187 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:12:02.660188 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:12:02.660260 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:12:02.660644 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:12:02.660645 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:12:02.660645 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:12:02.660807 (+ 162us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:12:02.660857 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0621 05:12:02.661326 (+ 469us) server_negotiation.cc:299] Negotiation successful
0621 05:12:02.661431 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260621 05:12:02.662456 15311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:12:02.648242 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.131.190:35685 (local address 127.3.131.131:41531)
0621 05:12:02.648423 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0621 05:12:02.648473 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0621 05:12:02.649181 (+ 708us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:12:02.649580 (+ 399us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:12:02.649585 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:12:02.649867 (+ 282us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:12:02.650167 (+ 300us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:12:02.650174 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:12:02.651246 (+ 1072us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:12:02.651247 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:12:02.652580 (+ 1333us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:12:02.652583 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:12:02.653167 (+ 584us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:12:02.653168 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:12:02.653331 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:12:02.654107 (+ 776us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 05:12:02.654118 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:12:02.657246 (+ 3128us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 05:12:02.659671 (+ 2425us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:12:02.659682 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:12:02.659686 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:12:02.660072 (+ 386us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:12:02.660377 (+ 305us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:12:02.660378 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:12:02.660378 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:12:02.660548 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:12:02.660911 (+ 363us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 05:12:02.660917 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 05:12:02.661106 (+ 189us) client_negotiation.cc:769] Sending connection context
0621 05:12:02.661979 (+ 873us) client_negotiation.cc:240] Negotiation successful
0621 05:12:02.662181 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":94,"threads_started":1}
I20260621 05:12:02.663388 15309 heartbeater.cc:347] Connected to a master server at 127.3.131.190:35685
I20260621 05:12:02.663604 15309 heartbeater.cc:464] Registering TS with master...
I20260621 05:12:02.663862 15309 heartbeater.cc:511] Master 127.3.131.190:35685 requested a full tablet report, sending...
I20260621 05:12:02.664368 14825 ts_manager.cc:194] Registered new tserver with Master: 78d6b458616b46e28882ddfc019c3dee (127.3.131.131:33053)
I20260621 05:12:02.664884 14825 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.131.131@KRBTEST.COM'} at 127.3.131.131:41531
I20260621 05:12:02.669447 3598 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260621 05:12:02.680310 15040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:12:02.671518 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40644 (local address 127.3.131.190:35685)
0621 05:12:02.671564 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0621 05:12:02.671566 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:12:02.671708 (+ 142us) server_negotiation.cc:407] Connection header received
0621 05:12:02.671761 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:12:02.671762 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:12:02.671772 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:12:02.671845 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:12:02.672618 (+ 773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:12:02.673403 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:12:02.674686 (+ 1283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:12:02.674982 (+ 296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:12:02.675085 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:12:02.676876 (+ 1791us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:12:02.676878 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:12:02.676878 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:12:02.676887 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:12:02.678777 (+ 1890us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:12:02.679328 (+ 551us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:12:02.679329 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:12:02.679329 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:12:02.679392 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:12:02.679597 (+ 205us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:12:02.679598 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:12:02.679598 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:12:02.679735 (+ 137us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:12:02.679811 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0621 05:12:02.680108 (+ 297us) server_negotiation.cc:299] Negotiation successful
0621 05:12:02.680209 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260621 05:12:02.682648 14825 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40644:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260621 05:12:02.685236 14825 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260621 05:12:02.696744 15322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:12:02.688706 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42574 (local address 127.3.131.131:33053)
0621 05:12:02.691789 (+ 3083us) server_negotiation.cc:206] Beginning negotiation
0621 05:12:02.691793 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 05:12:02.691804 (+ 11us) server_negotiation.cc:407] Connection header received
0621 05:12:02.691846 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:12:02.691848 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:12:02.691865 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:12:02.691947 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 05:12:02.692394 (+ 447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:12:02.693495 (+ 1101us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:12:02.695124 (+ 1629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:12:02.696098 (+ 974us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:12:02.696497 (+ 399us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:12:02.696584 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0621 05:12:02.696612 (+ 28us) server_negotiation.cc:299] Negotiation successful
0621 05:12:02.696655 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3041,"thread_start_us":80,"threads_started":1}
I20260621 05:12:02.696624 15321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:12:02.688940 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.131.129:39409 (local address 127.0.0.1:51392)
0621 05:12:02.691598 (+ 2658us) negotiation.cc:107] Waiting for socket to connect
0621 05:12:02.691607 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0621 05:12:02.691660 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:12:02.692154 (+ 494us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:12:02.692154 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:12:02.692157 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 05:12:02.692206 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:12:02.692207 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:12:02.693535 (+ 1328us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:12:02.693536 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:12:02.695239 (+ 1703us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:12:02.695242 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:12:02.696297 (+ 1055us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:12:02.696298 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:12:02.696383 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:12:02.696397 (+ 14us) client_negotiation.cc:769] Sending connection context
0621 05:12:02.696467 (+ 70us) client_negotiation.cc:240] Negotiation successful
0621 05:12:02.696531 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2638,"thread_start_us":87,"threads_started":1}
I20260621 05:12:02.697283 15325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:12:02.688434 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.131.131:33053 (local address 127.0.0.1:42574)
0621 05:12:02.691334 (+ 2900us) negotiation.cc:107] Waiting for socket to connect
0621 05:12:02.691393 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0621 05:12:02.691475 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:12:02.692058 (+ 583us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:12:02.692062 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:12:02.692064 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 05:12:02.692154 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:12:02.692156 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:12:02.693599 (+ 1443us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:12:02.693600 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:12:02.694993 (+ 1393us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:12:02.694997 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:12:02.696281 (+ 1284us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:12:02.696282 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:12:02.696387 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:12:02.696400 (+ 13us) client_negotiation.cc:769] Sending connection context
0621 05:12:02.696468 (+ 68us) client_negotiation.cc:240] Negotiation successful
0621 05:12:02.697181 (+ 713us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2849,"mutex_wait_us":741,"thread_start_us":84,"threads_started":1}
I20260621 05:12:02.697960 15324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:12:02.689031 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51392 (local address 127.3.131.129:39409)
0621 05:12:02.691342 (+ 2311us) server_negotiation.cc:206] Beginning negotiation
0621 05:12:02.691348 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 05:12:02.691877 (+ 529us) server_negotiation.cc:407] Connection header received
0621 05:12:02.691920 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:12:02.691922 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:12:02.691940 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:12:02.692075 (+ 135us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 05:12:02.692277 (+ 202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:12:02.693434 (+ 1157us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:12:02.695334 (+ 1900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:12:02.696120 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:12:02.696194 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:12:02.696262 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0621 05:12:02.697814 (+ 1552us) server_negotiation.cc:299] Negotiation successful
0621 05:12:02.697869 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2269,"thread_start_us":50,"threads_started":1}
I20260621 05:12:02.698279 15229 tablet_service.cc:1467] Processing CreateTablet for tablet 2fad2c63069d4896ba35ef97174f3374 (DEFAULT_TABLE table=test-table [id=732a4f91c1bb4e659086ae620a869d0e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 05:12:02.698707 15229 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2fad2c63069d4896ba35ef97174f3374. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 05:12:02.698726 14971 tablet_service.cc:1467] Processing CreateTablet for tablet 2fad2c63069d4896ba35ef97174f3374 (DEFAULT_TABLE table=test-table [id=732a4f91c1bb4e659086ae620a869d0e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 05:12:02.699085 14971 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2fad2c63069d4896ba35ef97174f3374. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 05:12:02.702713 15327 tablet_bootstrap.cc:492] T 2fad2c63069d4896ba35ef97174f3374 P c51448aadc654a55bd7162d5b990579f: Bootstrap starting.
I20260621 05:12:02.703085 15328 tablet_bootstrap.cc:492] T 2fad2c63069d4896ba35ef97174f3374 P 78d6b458616b46e28882ddfc019c3dee: Bootstrap starting.
I20260621 05:12:02.703613 15327 tablet_bootstrap.cc:654] T 2fad2c63069d4896ba35ef97174f3374 P c51448aadc654a55bd7162d5b990579f: Neither blocks nor log segments found. Creating new log.
I20260621 05:12:02.704097 15327 log.cc:824] T 2fad2c63069d4896ba35ef97174f3374 P c51448aadc654a55bd7162d5b990579f: Log is configured to *not* fsync() on all Append() calls
I20260621 05:12:02.704324 15328 tablet_bootstrap.cc:654] T 2fad2c63069d4896ba35ef97174f3374 P 78d6b458616b46e28882ddfc019c3dee: Neither blocks nor log segments found. Creating new log.
I20260621 05:12:02.704685 15328 log.cc:824] T 2fad2c63069d4896ba35ef97174f3374 P 78d6b458616b46e28882ddfc019c3dee: Log is configured to *not* fsync() on all Append() calls
I20260621 05:12:02.705533 15327 tablet_bootstrap.cc:492] T 2fad2c63069d4896ba35ef97174f3374 P c51448aadc654a55bd7162d5b990579f: No bootstrap required, opened a new log
I20260621 05:12:02.705610 15327 ts_tablet_manager.cc:1397] T 2fad2c63069d4896ba35ef97174f3374 P c51448aadc654a55bd7162d5b990579f: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20260621 05:12:02.708124 15323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:12:02.689133 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.131.130:46525 (local address 127.0.0.1:38836)
0621 05:12:02.692255 (+ 3122us) negotiation.cc:107] Waiting for socket to connect
0621 05:12:02.692264 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0621 05:12:02.692323 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:12:02.703229 (+ 10906us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:12:02.703231 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:12:02.703235 (+ 4us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 05:12:02.703304 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:12:02.703306 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:12:02.704883 (+ 1577us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:12:02.704885 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:12:02.706230 (+ 1345us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:12:02.706235 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:12:02.707551 (+ 1316us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:12:02.707553 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:12:02.707744 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:12:02.707754 (+ 10us) client_negotiation.cc:769] Sending connection context
0621 05:12:02.707796 (+ 42us) client_negotiation.cc:240] Negotiation successful
0621 05:12:02.707861 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3107,"thread_start_us":55,"threads_started":1}
I20260621 05:12:02.709350 15326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:12:02.689325 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38836 (local address 127.3.131.130:46525)
0621 05:12:02.689943 (+ 618us) server_negotiation.cc:206] Beginning negotiation
0621 05:12:02.689949 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 05:12:02.702982 (+ 13033us) server_negotiation.cc:407] Connection header received
0621 05:12:02.703046 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:12:02.703049 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:12:02.703070 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:12:02.703389 (+ 319us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 05:12:02.703497 (+ 108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:12:02.704753 (+ 1256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:12:02.706447 (+ 1694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:12:02.707404 (+ 957us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:12:02.708625 (+ 1221us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:12:02.708761 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0621 05:12:02.708800 (+ 39us) server_negotiation.cc:299] Negotiation successful
0621 05:12:02.708858 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":575,"thread_start_us":81,"threads_started":1}
I20260621 05:12:02.710280 15327 raft_consensus.cc:348] T 2fad2c63069d4896ba35ef97174f3374 P c51448aadc654a55bd7162d5b990579f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c51448aadc654a55bd7162d5b990579f" member_type: VOTER last_known_addr { host: "127.3.131.129" port: 39409 } } peers { permanent_uuid: "3e5d83cff1c440d6929fa7475977853f" member_type: VOTER last_known_addr { host: "127.3.131.130" port: 46525 } } peers { permanent_uuid: "78d6b458616b46e28882ddfc019c3dee" member_type: VOTER last_known_addr { host: "127.3.131.131" port: 33053 } }
I20260621 05:12:02.710405 15109 tablet_service.cc:1467] Processing CreateTablet for tablet 2fad2c63069d4896ba35ef97174f3374 (DEFAULT_TABLE table=test-table [id=732a4f91c1bb4e659086ae620a869d0e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 05:12:02.710597 15327 raft_consensus.cc:374] T 2fad2c63069d4896ba35ef97174f3374 P c51448aadc654a55bd7162d5b990579f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 05:12:02.710649 15327 raft_consensus.cc:729] T 2fad2c63069d4896ba35ef97174f3374 P c51448aadc654a55bd7162d5b990579f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c51448aadc654a55bd7162d5b990579f, State: Initialized, Role: FOLLOWER
I20260621 05:12:02.710850 15109 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2fad2c63069d4896ba35ef97174f3374. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 05:12:02.710820 15327 consensus_queue.cc:260] T 2fad2c63069d4896ba35ef97174f3374 P c51448aadc654a55bd7162d5b990579f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c51448aadc654a55bd7162d5b990579f" member_type: VOTER last_known_addr { host: "127.3.131.129" port: 39409 } } peers { permanent_uuid: "3e5d83cff1c440d6929fa7475977853f" member_type: VOTER last_known_addr { host: "127.3.131.130" port: 46525 } } peers { permanent_uuid: "78d6b458616b46e28882ddfc019c3dee" member_type: VOTER last_known_addr { host: "127.3.131.131" port: 33053 } }
I20260621 05:12:02.711194 15327 ts_tablet_manager.cc:1428] T 2fad2c63069d4896ba35ef97174f3374 P c51448aadc654a55bd7162d5b990579f: Time spent starting tablet: real 0.006s user 0.000s sys 0.005s
I20260621 05:12:02.711232 15328 tablet_bootstrap.cc:492] T 2fad2c63069d4896ba35ef97174f3374 P 78d6b458616b46e28882ddfc019c3dee: No bootstrap required, opened a new log
I20260621 05:12:02.711309 15328 ts_tablet_manager.cc:1397] T 2fad2c63069d4896ba35ef97174f3374 P 78d6b458616b46e28882ddfc019c3dee: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.000s
I20260621 05:12:02.713505 15328 raft_consensus.cc:348] T 2fad2c63069d4896ba35ef97174f3374 P 78d6b458616b46e28882ddfc019c3dee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c51448aadc654a55bd7162d5b990579f" member_type: VOTER last_known_addr { host: "127.3.131.129" port: 39409 } } peers { permanent_uuid: "3e5d83cff1c440d6929fa7475977853f" member_type: VOTER last_known_addr { host: "127.3.131.130" port: 46525 } } peers { permanent_uuid: "78d6b458616b46e28882ddfc019c3dee" member_type: VOTER last_known_addr { host: "127.3.131.131" port: 33053 } }
I20260621 05:12:02.713701 15328 raft_consensus.cc:374] T 2fad2c63069d4896ba35ef97174f3374 P 78d6b458616b46e28882ddfc019c3dee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 05:12:02.713742 15328 raft_consensus.cc:729] T 2fad2c63069d4896ba35ef97174f3374 P 78d6b458616b46e28882ddfc019c3dee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 78d6b458616b46e28882ddfc019c3dee, State: Initialized, Role: FOLLOWER
I20260621 05:12:02.713907 15328 consensus_queue.cc:260] T 2fad2c63069d4896ba35ef97174f3374 P 78d6b458616b46e28882ddfc019c3dee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c51448aadc654a55bd7162d5b990579f" member_type: VOTER last_known_addr { host: "127.3.131.129" port: 39409 } } peers { permanent_uuid: "3e5d83cff1c440d6929fa7475977853f" member_type: VOTER last_known_addr { host: "127.3.131.130" port: 46525 } } peers { permanent_uuid: "78d6b458616b46e28882ddfc019c3dee" member_type: VOTER last_known_addr { host: "127.3.131.131" port: 33053 } }
I20260621 05:12:02.714224 15328 ts_tablet_manager.cc:1428] T 2fad2c63069d4896ba35ef97174f3374 P 78d6b458616b46e28882ddfc019c3dee: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260621 05:12:02.714366 15309 heartbeater.cc:503] Master 127.3.131.190:35685 was elected leader, sending a full tablet report...
I20260621 05:12:02.715165 15037 heartbeater.cc:503] Master 127.3.131.190:35685 was elected leader, sending a full tablet report...
I20260621 05:12:02.716255 15333 tablet_bootstrap.cc:492] T 2fad2c63069d4896ba35ef97174f3374 P 3e5d83cff1c440d6929fa7475977853f: Bootstrap starting.
I20260621 05:12:02.718918 15333 tablet_bootstrap.cc:654] T 2fad2c63069d4896ba35ef97174f3374 P 3e5d83cff1c440d6929fa7475977853f: Neither blocks nor log segments found. Creating new log.
I20260621 05:12:02.719477 15333 log.cc:824] T 2fad2c63069d4896ba35ef97174f3374 P 3e5d83cff1c440d6929fa7475977853f: Log is configured to *not* fsync() on all Append() calls
I20260621 05:12:02.720419 15333 tablet_bootstrap.cc:492] T 2fad2c63069d4896ba35ef97174f3374 P 3e5d83cff1c440d6929fa7475977853f: No bootstrap required, opened a new log
I20260621 05:12:02.720530 15333 ts_tablet_manager.cc:1397] T 2fad2c63069d4896ba35ef97174f3374 P 3e5d83cff1c440d6929fa7475977853f: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.002s
I20260621 05:12:02.722241 15333 raft_consensus.cc:348] T 2fad2c63069d4896ba35ef97174f3374 P 3e5d83cff1c440d6929fa7475977853f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c51448aadc654a55bd7162d5b990579f" member_type: VOTER last_known_addr { host: "127.3.131.129" port: 39409 } } peers { permanent_uuid: "3e5d83cff1c440d6929fa7475977853f" member_type: VOTER last_known_addr { host: "127.3.131.130" port: 46525 } } peers { permanent_uuid: "78d6b458616b46e28882ddfc019c3dee" member_type: VOTER last_known_addr { host: "127.3.131.131" port: 33053 } }
I20260621 05:12:02.722436 15333 raft_consensus.cc:374] T 2fad2c63069d4896ba35ef97174f3374 P 3e5d83cff1c440d6929fa7475977853f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 05:12:02.722479 15333 raft_consensus.cc:729] T 2fad2c63069d4896ba35ef97174f3374 P 3e5d83cff1c440d6929fa7475977853f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3e5d83cff1c440d6929fa7475977853f, State: Initialized, Role: FOLLOWER
I20260621 05:12:02.722623 15333 consensus_queue.cc:260] T 2fad2c63069d4896ba35ef97174f3374 P 3e5d83cff1c440d6929fa7475977853f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c51448aadc654a55bd7162d5b990579f" member_type: VOTER last_known_addr { host: "127.3.131.129" port: 39409 } } peers { permanent_uuid: "3e5d83cff1c440d6929fa7475977853f" member_type: VOTER last_known_addr { host: "127.3.131.130" port: 46525 } } peers { permanent_uuid: "78d6b458616b46e28882ddfc019c3dee" member_type: VOTER last_known_addr { host: "127.3.131.131" port: 33053 } }
I20260621 05:12:02.722986 15333 ts_tablet_manager.cc:1428] T 2fad2c63069d4896ba35ef97174f3374 P 3e5d83cff1c440d6929fa7475977853f: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20260621 05:12:02.723019 15175 heartbeater.cc:503] Master 127.3.131.190:35685 was elected leader, sending a full tablet report...
Jun 21 05:12:02 dist-test-slave-vrq1 krb5kdc[14349](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782018722, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.131.190@KRBTEST.COM for HTTP/127.3.131.148@KRBTEST.COM
W20260621 05:12:02.759526 15038 tablet.cc:2369] T 2fad2c63069d4896ba35ef97174f3374 P c51448aadc654a55bd7162d5b990579f: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260621 05:12:02.827652 15176 tablet.cc:2369] T 2fad2c63069d4896ba35ef97174f3374 P 3e5d83cff1c440d6929fa7475977853f: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260621 05:12:02.905818 15310 tablet.cc:2369] T 2fad2c63069d4896ba35ef97174f3374 P 78d6b458616b46e28882ddfc019c3dee: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260621 05:12:02.972543 15335 raft_consensus.cc:482] T 2fad2c63069d4896ba35ef97174f3374 P 3e5d83cff1c440d6929fa7475977853f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260621 05:12:02.972689 15335 raft_consensus.cc:504] T 2fad2c63069d4896ba35ef97174f3374 P 3e5d83cff1c440d6929fa7475977853f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c51448aadc654a55bd7162d5b990579f" member_type: VOTER last_known_addr { host: "127.3.131.129" port: 39409 } } peers { permanent_uuid: "3e5d83cff1c440d6929fa7475977853f" member_type: VOTER last_known_addr { host: "127.3.131.130" port: 46525 } } peers { permanent_uuid: "78d6b458616b46e28882ddfc019c3dee" member_type: VOTER last_known_addr { host: "127.3.131.131" port: 33053 } }
I20260621 05:12:02.973104 15335 leader_election.cc:288] T 2fad2c63069d4896ba35ef97174f3374 P 3e5d83cff1c440d6929fa7475977853f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c51448aadc654a55bd7162d5b990579f (127.3.131.129:39409), 78d6b458616b46e28882ddfc019c3dee (127.3.131.131:33053)
I20260621 05:12:02.978399 15324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:12:02.973926 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.131.130:41619 (local address 127.3.131.129:39409)
0621 05:12:02.974049 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0621 05:12:02.974051 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:12:02.974175 (+ 124us) server_negotiation.cc:407] Connection header received
0621 05:12:02.974287 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:12:02.974289 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:12:02.974299 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:12:02.974330 (+ 31us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 05:12:02.974558 (+ 228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:12:02.975320 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:12:02.976926 (+ 1606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:12:02.977536 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:12:02.977602 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:12:02.977612 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0621 05:12:02.978275 (+ 663us) server_negotiation.cc:299] Negotiation successful
0621 05:12:02.978323 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":100}
I20260621 05:12:02.978849 15336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:12:02.973405 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.131.129:39409 (local address 127.3.131.130:41619)
0621 05:12:02.974100 (+ 695us) negotiation.cc:107] Waiting for socket to connect
0621 05:12:02.974112 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0621 05:12:02.974204 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:12:02.974405 (+ 201us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:12:02.974406 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:12:02.974408 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 05:12:02.974475 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:12:02.974478 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:12:02.975437 (+ 959us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:12:02.975438 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:12:02.976824 (+ 1386us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:12:02.976827 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:12:02.978100 (+ 1273us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:12:02.978100 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:12:02.978178 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:12:02.978181 (+ 3us) client_negotiation.cc:769] Sending connection context
0621 05:12:02.978721 (+ 540us) client_negotiation.cc:240] Negotiation successful
0621 05:12:02.978767 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":680,"thread_start_us":75,"threads_started":1}
I20260621 05:12:02.979337 14991 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2fad2c63069d4896ba35ef97174f3374" candidate_uuid: "3e5d83cff1c440d6929fa7475977853f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c51448aadc654a55bd7162d5b990579f" is_pre_election: true
I20260621 05:12:02.979519 14991 raft_consensus.cc:2454] T 2fad2c63069d4896ba35ef97174f3374 P c51448aadc654a55bd7162d5b990579f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3e5d83cff1c440d6929fa7475977853f in term 0.
I20260621 05:12:02.979904 15062 leader_election.cc:302] T 2fad2c63069d4896ba35ef97174f3374 P 3e5d83cff1c440d6929fa7475977853f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3e5d83cff1c440d6929fa7475977853f, c51448aadc654a55bd7162d5b990579f; no voters:
I20260621 05:12:02.980047 15335 raft_consensus.cc:2781] T 2fad2c63069d4896ba35ef97174f3374 P 3e5d83cff1c440d6929fa7475977853f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260621 05:12:02.980085 15335 raft_consensus.cc:482] T 2fad2c63069d4896ba35ef97174f3374 P 3e5d83cff1c440d6929fa7475977853f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260621 05:12:02.980106 15335 raft_consensus.cc:3037] T 2fad2c63069d4896ba35ef97174f3374 P 3e5d83cff1c440d6929fa7475977853f [term 0 FOLLOWER]: Advancing to term 1
I20260621 05:12:02.981015 15335 raft_consensus.cc:504] T 2fad2c63069d4896ba35ef97174f3374 P 3e5d83cff1c440d6929fa7475977853f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c51448aadc654a55bd7162d5b990579f" member_type: VOTER last_known_addr { host: "127.3.131.129" port: 39409 } } peers { permanent_uuid: "3e5d83cff1c440d6929fa7475977853f" member_type: VOTER last_known_addr { host: "127.3.131.130" port: 46525 } } peers { permanent_uuid: "78d6b458616b46e28882ddfc019c3dee" member_type: VOTER last_known_addr { host: "127.3.131.131" port: 33053 } }
I20260621 05:12:02.981150 15335 leader_election.cc:288] T 2fad2c63069d4896ba35ef97174f3374 P 3e5d83cff1c440d6929fa7475977853f [CANDIDATE]: Term 1 election: Requested vote from peers c51448aadc654a55bd7162d5b990579f (127.3.131.129:39409), 78d6b458616b46e28882ddfc019c3dee (127.3.131.131:33053)
I20260621 05:12:02.981319 15322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:12:02.973995 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.131.130:34703 (local address 127.3.131.131:33053)
0621 05:12:02.974044 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0621 05:12:02.974046 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:12:02.977828 (+ 3782us) server_negotiation.cc:407] Connection header received
0621 05:12:02.977935 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:12:02.977936 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:12:02.977950 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:12:02.977981 (+ 31us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 05:12:02.978228 (+ 247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:12:02.978919 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:12:02.980301 (+ 1382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:12:02.980896 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:12:02.980949 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:12:02.980958 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0621 05:12:02.981181 (+ 223us) server_negotiation.cc:299] Negotiation successful
0621 05:12:02.981243 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260621 05:12:02.981628 15337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:12:02.973627 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.131.131:33053 (local address 127.3.131.130:34703)
0621 05:12:02.977756 (+ 4129us) negotiation.cc:107] Waiting for socket to connect
0621 05:12:02.977769 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0621 05:12:02.977859 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:12:02.978050 (+ 191us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:12:02.978051 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:12:02.978052 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 05:12:02.978141 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:12:02.978142 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:12:02.979046 (+ 904us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:12:02.979046 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:12:02.980190 (+ 1144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:12:02.980193 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:12:02.981024 (+ 831us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:12:02.981025 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:12:02.981094 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:12:02.981098 (+ 4us) client_negotiation.cc:769] Sending connection context
0621 05:12:02.981508 (+ 410us) client_negotiation.cc:240] Negotiation successful
0621 05:12:02.981553 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4111,"thread_start_us":62,"threads_started":1}
I20260621 05:12:02.981848 14991 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2fad2c63069d4896ba35ef97174f3374" candidate_uuid: "3e5d83cff1c440d6929fa7475977853f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c51448aadc654a55bd7162d5b990579f"
I20260621 05:12:02.981943 14991 raft_consensus.cc:3037] T 2fad2c63069d4896ba35ef97174f3374 P c51448aadc654a55bd7162d5b990579f [term 0 FOLLOWER]: Advancing to term 1
I20260621 05:12:02.982169 15263 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2fad2c63069d4896ba35ef97174f3374" candidate_uuid: "3e5d83cff1c440d6929fa7475977853f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78d6b458616b46e28882ddfc019c3dee" is_pre_election: true
I20260621 05:12:02.982307 15263 raft_consensus.cc:2454] T 2fad2c63069d4896ba35ef97174f3374 P 78d6b458616b46e28882ddfc019c3dee [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3e5d83cff1c440d6929fa7475977853f in term 0.
I20260621 05:12:02.982440 15262 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2fad2c63069d4896ba35ef97174f3374" candidate_uuid: "3e5d83cff1c440d6929fa7475977853f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78d6b458616b46e28882ddfc019c3dee"
I20260621 05:12:02.982527 15262 raft_consensus.cc:3037] T 2fad2c63069d4896ba35ef97174f3374 P 78d6b458616b46e28882ddfc019c3dee [term 0 FOLLOWER]: Advancing to term 1
I20260621 05:12:02.983011 14991 raft_consensus.cc:2454] T 2fad2c63069d4896ba35ef97174f3374 P c51448aadc654a55bd7162d5b990579f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3e5d83cff1c440d6929fa7475977853f in term 1.
I20260621 05:12:02.983271 15062 leader_election.cc:302] T 2fad2c63069d4896ba35ef97174f3374 P 3e5d83cff1c440d6929fa7475977853f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3e5d83cff1c440d6929fa7475977853f, c51448aadc654a55bd7162d5b990579f; no voters:
I20260621 05:12:02.983415 15335 raft_consensus.cc:2781] T 2fad2c63069d4896ba35ef97174f3374 P 3e5d83cff1c440d6929fa7475977853f [term 1 FOLLOWER]: Leader election won for term 1
I20260621 05:12:02.983566 15335 raft_consensus.cc:686] T 2fad2c63069d4896ba35ef97174f3374 P 3e5d83cff1c440d6929fa7475977853f [term 1 LEADER]: Becoming Leader. State: Replica: 3e5d83cff1c440d6929fa7475977853f, State: Running, Role: LEADER
I20260621 05:12:02.983645 15335 consensus_queue.cc:237] T 2fad2c63069d4896ba35ef97174f3374 P 3e5d83cff1c440d6929fa7475977853f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c51448aadc654a55bd7162d5b990579f" member_type: VOTER last_known_addr { host: "127.3.131.129" port: 39409 } } peers { permanent_uuid: "3e5d83cff1c440d6929fa7475977853f" member_type: VOTER last_known_addr { host: "127.3.131.130" port: 46525 } } peers { permanent_uuid: "78d6b458616b46e28882ddfc019c3dee" member_type: VOTER last_known_addr { host: "127.3.131.131" port: 33053 } }
I20260621 05:12:02.983713 15262 raft_consensus.cc:2454] T 2fad2c63069d4896ba35ef97174f3374 P 78d6b458616b46e28882ddfc019c3dee [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3e5d83cff1c440d6929fa7475977853f in term 1.
I20260621 05:12:02.984431 14824 catalog_manager.cc:5495] T 2fad2c63069d4896ba35ef97174f3374 P 3e5d83cff1c440d6929fa7475977853f reported cstate change: term changed from 0 to 1, leader changed from <none> to 3e5d83cff1c440d6929fa7475977853f (127.3.131.130). New cstate: current_term: 1 leader_uuid: "3e5d83cff1c440d6929fa7475977853f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c51448aadc654a55bd7162d5b990579f" member_type: VOTER last_known_addr { host: "127.3.131.129" port: 39409 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3e5d83cff1c440d6929fa7475977853f" member_type: VOTER last_known_addr { host: "127.3.131.130" port: 46525 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "78d6b458616b46e28882ddfc019c3dee" member_type: VOTER last_known_addr { host: "127.3.131.131" port: 33053 } health_report { overall_health: UNKNOWN } } }
I20260621 05:12:03.055099 15326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:12:03.051383 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38846 (local address 127.3.131.130:46525)
0621 05:12:03.051434 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0621 05:12:03.051436 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:12:03.051446 (+ 10us) server_negotiation.cc:407] Connection header received
0621 05:12:03.051508 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:12:03.051509 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:12:03.051529 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:12:03.051631 (+ 102us) server_negotiation.cc:226] Negotiated authn=TOKEN
0621 05:12:03.051778 (+ 147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:12:03.052508 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:12:03.053825 (+ 1317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:12:03.054131 (+ 306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:12:03.054200 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:12:03.054412 (+ 212us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0621 05:12:03.054572 (+ 160us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0621 05:12:03.054656 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0621 05:12:03.054917 (+ 261us) server_negotiation.cc:299] Negotiation successful
0621 05:12:03.054941 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260621 05:12:03.057672 14991 raft_consensus.cc:1261] T 2fad2c63069d4896ba35ef97174f3374 P c51448aadc654a55bd7162d5b990579f [term 1 FOLLOWER]: Refusing update from remote peer 3e5d83cff1c440d6929fa7475977853f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 05:12:03.057678 15262 raft_consensus.cc:1261] T 2fad2c63069d4896ba35ef97174f3374 P 78d6b458616b46e28882ddfc019c3dee [term 1 FOLLOWER]: Refusing update from remote peer 3e5d83cff1c440d6929fa7475977853f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 05:12:03.058121 15338 consensus_queue.cc:1035] T 2fad2c63069d4896ba35ef97174f3374 P 3e5d83cff1c440d6929fa7475977853f [LEADER]: Connected to new peer: Peer: permanent_uuid: "78d6b458616b46e28882ddfc019c3dee" member_type: VOTER last_known_addr { host: "127.3.131.131" port: 33053 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 05:12:03.058332 15335 consensus_queue.cc:1035] T 2fad2c63069d4896ba35ef97174f3374 P 3e5d83cff1c440d6929fa7475977853f [LEADER]: Connected to new peer: Peer: permanent_uuid: "c51448aadc654a55bd7162d5b990579f" member_type: VOTER last_known_addr { host: "127.3.131.129" port: 39409 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 05:12:03.061506 15341 mvcc.cc:205] Tried to move back new op lower bound from 7299148689641398272 to 7299148689341460480. Current Snapshot: MvccSnapshot[applied={T|T < 7299148689641398272}]
I20260621 05:12:03.063306 15343 mvcc.cc:205] Tried to move back new op lower bound from 7299148689641398272 to 7299148689341460480. Current Snapshot: MvccSnapshot[applied={T|T < 7299148689641398272}]
I20260621 05:12:03.070619 14824 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40644:
table { table_name: "test-table" } modify_external_catalogs: true
I20260621 05:12:03.070739 14824 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40644:
table { table_name: "test-table" } modify_external_catalogs: true
I20260621 05:12:03.071939 14824 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 8c591b6c189f45efad224430a2e55fcb: Sending DeleteTablet for 3 replicas of tablet 2fad2c63069d4896ba35ef97174f3374
I20260621 05:12:03.072376 15229 tablet_service.cc:1514] Processing DeleteTablet for tablet 2fad2c63069d4896ba35ef97174f3374 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 05:12:03 UTC) from {username='kudu', principal='kudu/127.3.131.190@KRBTEST.COM'} at 127.0.0.1:42574
I20260621 05:12:03.072786 15109 tablet_service.cc:1514] Processing DeleteTablet for tablet 2fad2c63069d4896ba35ef97174f3374 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 05:12:03 UTC) from {username='kudu', principal='kudu/127.3.131.190@KRBTEST.COM'} at 127.0.0.1:38836
I20260621 05:12:03.073125 15351 tablet_replica.cc:330] T 2fad2c63069d4896ba35ef97174f3374 P 78d6b458616b46e28882ddfc019c3dee: stopping tablet replica
I20260621 05:12:03.073117 14971 tablet_service.cc:1514] Processing DeleteTablet for tablet 2fad2c63069d4896ba35ef97174f3374 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 05:12:03 UTC) from {username='kudu', principal='kudu/127.3.131.190@KRBTEST.COM'} at 127.0.0.1:51392
I20260621 05:12:03.073264 15351 raft_consensus.cc:2229] T 2fad2c63069d4896ba35ef97174f3374 P 78d6b458616b46e28882ddfc019c3dee [term 1 FOLLOWER]: Raft consensus shutting down.
I20260621 05:12:03.073338 15351 raft_consensus.cc:2258] T 2fad2c63069d4896ba35ef97174f3374 P 78d6b458616b46e28882ddfc019c3dee [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 05:12:03.073580 15345 mvcc.cc:205] Tried to move back new op lower bound from 7299148689641398272 to 7299148689341460480. Current Snapshot: MvccSnapshot[applied={T|T < 7299148689641398272}]
I20260621 05:12:03.075029 15351 ts_tablet_manager.cc:1905] T 2fad2c63069d4896ba35ef97174f3374 P 78d6b458616b46e28882ddfc019c3dee: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 05:12:03.075461 3598 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWZOx2_/build/release/bin/kudu with pid 14906
I20260621 05:12:03.077411 15352 tablet_replica.cc:330] T 2fad2c63069d4896ba35ef97174f3374 P 3e5d83cff1c440d6929fa7475977853f: stopping tablet replica
I20260621 05:12:03.077535 15352 raft_consensus.cc:2229] T 2fad2c63069d4896ba35ef97174f3374 P 3e5d83cff1c440d6929fa7475977853f [term 1 LEADER]: Raft consensus shutting down.
I20260621 05:12:03.077651 15352 raft_consensus.cc:2258] T 2fad2c63069d4896ba35ef97174f3374 P 3e5d83cff1c440d6929fa7475977853f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 05:12:03.078521 15351 ts_tablet_manager.cc:1918] T 2fad2c63069d4896ba35ef97174f3374 P 78d6b458616b46e28882ddfc019c3dee: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260621 05:12:03.078591 15351 log.cc:1196] T 2fad2c63069d4896ba35ef97174f3374 P 78d6b458616b46e28882ddfc019c3dee: Deleting WAL directory at /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782018616648578-3598-0/minicluster-data/ts-2/wal/wals/2fad2c63069d4896ba35ef97174f3374
I20260621 05:12:03.078866 15351 ts_tablet_manager.cc:1939] T 2fad2c63069d4896ba35ef97174f3374 P 78d6b458616b46e28882ddfc019c3dee: Deleting consensus metadata
I20260621 05:12:03.080601 15352 ts_tablet_manager.cc:1905] T 2fad2c63069d4896ba35ef97174f3374 P 3e5d83cff1c440d6929fa7475977853f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 05:12:03.082405 15352 ts_tablet_manager.cc:1918] T 2fad2c63069d4896ba35ef97174f3374 P 3e5d83cff1c440d6929fa7475977853f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260621 05:12:03.082479 15352 log.cc:1196] T 2fad2c63069d4896ba35ef97174f3374 P 3e5d83cff1c440d6929fa7475977853f: Deleting WAL directory at /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782018616648578-3598-0/minicluster-data/ts-1/wal/wals/2fad2c63069d4896ba35ef97174f3374
I20260621 05:12:03.082734 15352 ts_tablet_manager.cc:1939] T 2fad2c63069d4896ba35ef97174f3374 P 3e5d83cff1c440d6929fa7475977853f: Deleting consensus metadata
I20260621 05:12:03.083375 14810 catalog_manager.cc:4841] TS 3e5d83cff1c440d6929fa7475977853f (127.3.131.130:46525): tablet 2fad2c63069d4896ba35ef97174f3374 (table test-table [id=732a4f91c1bb4e659086ae620a869d0e]) successfully deleted
W20260621 05:12:03.089401 14810 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.3.131.129:39409 (error 108)
I20260621 05:12:03.090128 3598 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWZOx2_/build/release/bin/kudu with pid 15043
I20260621 05:12:03.092829 14809 catalog_manager.cc:4841] TS 78d6b458616b46e28882ddfc019c3dee (127.3.131.131:33053): tablet 2fad2c63069d4896ba35ef97174f3374 (table test-table [id=732a4f91c1bb4e659086ae620a869d0e]) successfully deleted
I20260621 05:12:03.093901 15323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:12:03.093537 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.131.129:39409 (local address 127.0.0.1:51400)
0621 05:12:03.093794 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0621 05:12:03.093857 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.131.129:39409: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":217}
Jun 21 05:12:03 dist-test-slave-vrq1 krb5kdc[14349](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782018722, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.131.190@KRBTEST.COM for HTTP/127.3.131.148@KRBTEST.COM
W20260621 05:12:03.094164 14810 catalog_manager.cc:4584] TS c51448aadc654a55bd7162d5b990579f (127.3.131.129:39409): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 2fad2c63069d4896ba35ef97174f3374: Network error: Client connection negotiation failed: client connection to 127.3.131.129:39409: connect: Connection refused (error 111)
I20260621 05:12:03.098135 3598 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWZOx2_/build/release/bin/kudu with pid 15180
I20260621 05:12:03.107244 3598 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWZOx2_/build/release/bin/kudu with pid 14793
I20260621 05:12:03.115944 3598 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260621 05:12:03.461292 3598 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260621 05:12:03.461406 3598 mini_ranger.cc:66] Stopping Ranger...
I20260621 05:12:03.807252 3598 mini_ranger.cc:68] Stopped Ranger
2026-06-21 05:12:03.807 UTC [14379] LOG: received smart shutdown request
2026-06-21 05:12:03.812 UTC [14379] LOG: background worker "logical replication launcher" (PID 14388) exited with exit code 1
2026-06-21 05:12:03.812 UTC [14383] LOG: shutting down
2026-06-21 05:12:04.036 UTC [14379] LOG: database system is shut down
2026-06-21T05:12:04Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (49255 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782018616648578-3598-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 21 05:12:04 dist-test-slave-vrq1 krb5kdc[15364](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 21 05:12:04 dist-test-slave-vrq1 krb5kdc[15364](info): set up 2 sockets
Jun 21 05:12:04 dist-test-slave-vrq1 krb5kdc[15364](info): commencing operation
krb5kdc: starting...
W20260621 05:12:05.215801 3598 mini_kdc.cc:121] Time spent starting KDC: real 1.051s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782018616648578-3598-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782018616648578-3598-0/krb5kdc/test-user.keytab.
Jun 21 05:12:05 dist-test-slave-vrq1 krb5kdc[15364](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782018725, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-21T05:12:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-21T05:12:05Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.3.131.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.3.131.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.3.131.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782018616648578-3598-0/krb5kdc/rangeradmin_127.3.131.148@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.3.131.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782018616648578-3598-0/krb5kdc/rangeradmin_127.3.131.148@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.3.131.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.3.131.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.3.131.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782018616648578-3598-0/krb5kdc/rangerlookup_127.3.131.148@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.3.131.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782018616648578-3598-0/krb5kdc/rangerlookup_127.3.131.148@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.3.131.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.3.131.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.3.131.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782018616648578-3598-0/krb5kdc/HTTP_127.3.131.148@KRBTEST.COM.keytab.
Entry for principal HTTP/127.3.131.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782018616648578-3598-0/krb5kdc/HTTP_127.3.131.148@KRBTEST.COM.keytab.
I20260621 05:12:05.394130 3598 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782018616648578-3598-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskWZOx2_/build/release/bin/postgres/pg_ctl -D /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782018616648578-3598-0/minicluster-data/postgres -l logfile start
2026-06-21 05:12:08.422 UTC [15394] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-06-21 05:12:08.422 UTC [15394] LOG: listening on IPv4 address "127.3.131.148", port 44359
2026-06-21 05:12:08.427 UTC [15394] LOG: listening on Unix socket "/tmp/.s.PGSQL.44359"
2026-06-21 05:12:08.463 UTC [15397] LOG: database system was shut down at 2026-06-21 05:12:06 UTC
2026-06-21 05:12:08.468 UTC [15394] LOG: database system is ready to accept connections
I20260621 05:12:09.427300 3598 mini_postgres.cc:96] Postgres bound to 44359
2026-06-21 05:12:09.433 UTC [15405] FATAL: database "slave" does not exist
127.3.131.148:44359 - accepting connections
I20260621 05:12:09.434212 3598 mini_ranger.cc:161] Starting Ranger...
I20260621 05:12:09.450255 3598 mini_ranger.cc:84] Created miniranger Postgres user
I20260621 05:12:10.039289 3598 mini_ranger.cc:87] Created ranger Postgres database
I20260621 05:12:10.039378 3598 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782018616648578-3598-0/minicluster-data/ranger-admin
2026-06-21 05:12:10,340 [I] DB FLAVOR :POSTGRES
2026-06-21 05:12:10,341 [I] --------- Verifying Ranger DB connection ---------
2026-06-21 05:12:10,341 [I] Checking connection..
2026-06-21 05:12:10,341 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWZOx2_/build/release/bin/postgresql.jar:/tmp/dist-test-taskWZOx2_/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.131.148:44359/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-06-21 05:12:10,598 [I] Checking connection passed.
2026-06-21 05:12:10,599 [I] --------- Verifying version history table ---------
2026-06-21 05:12:10,599 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWZOx2_/build/release/bin/postgresql.jar:/tmp/dist-test-taskWZOx2_/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.131.148:44359/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-21 05:12:10,868 [I] Table x_db_version_h does not exist in database ranger
2026-06-21 05:12:10,868 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWZOx2_/build/release/bin/postgresql.jar:/tmp/dist-test-taskWZOx2_/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.131.148:44359/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-21 05:12:11,111 [I] Table x_db_version_h does not exist in database ranger
2026-06-21 05:12:11,111 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-06-21 05:12:11,111 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWZOx2_/build/release/bin/postgresql.jar:/tmp/dist-test-taskWZOx2_/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.131.148:44359/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskWZOx2_/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-06-21 05:12:11.342 UTC [15469] WARNING: there is no transaction in progress
2026-06-21 05:12:11,349 [I] create_dbversion_catalog.sql file imported successfully
2026-06-21 05:12:11,349 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWZOx2_/build/release/bin/postgresql.jar:/tmp/dist-test-taskWZOx2_/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.131.148:44359/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-21 05:12:11,621 [I] Table x_db_version_h already exists in database 'ranger'
2026-06-21 05:12:11,621 [I] --------- Importing Ranger Core DB Schema ---------
2026-06-21 05:12:11,621 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWZOx2_/build/release/bin/postgresql.jar:/tmp/dist-test-taskWZOx2_/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.131.148:44359/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-06-21 05:12:11,854 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWZOx2_/build/release/bin/postgresql.jar:/tmp/dist-test-taskWZOx2_/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.131.148:44359/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-06-21 05:12:12,134 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWZOx2_/build/release/bin/postgresql.jar:/tmp/dist-test-taskWZOx2_/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.131.148:44359/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-vrq1.c.gcp-upstream.internal','N') ;"
2026-06-21 05:12:12,363 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWZOx2_/build/release/bin/postgresql.jar:/tmp/dist-test-taskWZOx2_/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.131.148:44359/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-06-21 05:12:12,610 [I] Table x_portal_user does not exist in database ranger
2026-06-21 05:12:12,610 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWZOx2_/build/release/bin/postgresql.jar:/tmp/dist-test-taskWZOx2_/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.131.148:44359/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-06-21 05:12:12,886 [I] Table x_policy_ref_group does not exist in database ranger
2026-06-21 05:12:12,886 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWZOx2_/build/release/bin/postgresql.jar:/tmp/dist-test-taskWZOx2_/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.131.148:44359/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-06-21 05:12:13,124 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWZOx2_/build/release/bin/postgresql.jar:/tmp/dist-test-taskWZOx2_/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.131.148:44359/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-06-21 05:12:13,375 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-06-21 05:12:13,375 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWZOx2_/build/release/bin/postgresql.jar:/tmp/dist-test-taskWZOx2_/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.131.148:44359/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskWZOx2_/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-06-21 05:12:14.473 UTC [15577] WARNING: there is no transaction in progress
2026-06-21 05:12:14.492 UTC [15577] WARNING: there is no transaction in progress
2026-06-21 05:12:14.509 UTC [15577] WARNING: there is no transaction in progress
2026-06-21 05:12:14.527 UTC [15577] WARNING: there is no transaction in progress
2026-06-21 05:12:14.544 UTC [15577] WARNING: there is no transaction in progress
2026-06-21 05:12:14.566 UTC [15577] WARNING: there is no transaction in progress
2026-06-21 05:12:14.629 UTC [15577] WARNING: there is no transaction in progress
2026-06-21 05:12:14.637 UTC [15577] WARNING: there is no transaction in progress
2026-06-21 05:12:14.652 UTC [15577] WARNING: there is no transaction in progress
2026-06-21 05:12:14.663 UTC [15577] WARNING: there is no transaction in progress
2026-06-21 05:12:14.674 UTC [15577] WARNING: there is no transaction in progress
2026-06-21 05:12:14.689 UTC [15577] WARNING: there is no transaction in progress
2026-06-21 05:12:14.700 UTC [15577] WARNING: there is no transaction in progress
2026-06-21 05:12:14.707 UTC [15577] WARNING: there is no transaction in progress
2026-06-21 05:12:14.715 UTC [15577] WARNING: there is no transaction in progress
2026-06-21 05:12:14.723 UTC [15577] WARNING: there is no transaction in progress
2026-06-21 05:12:15,511 [I] ranger_core_db_postgres.sql file imported successfully
2026-06-21 05:12:15,511 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWZOx2_/build/release/bin/postgresql.jar:/tmp/dist-test-taskWZOx2_/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.131.148:44359/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-06-21 05:12:15,750 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-06-21 05:12:15,751 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWZOx2_/build/release/bin/postgresql.jar:/tmp/dist-test-taskWZOx2_/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.131.148:44359/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-06-21 05:12:16,004 [I] Table x_portal_user already exists in database 'ranger'
2026-06-21 05:12:16,005 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWZOx2_/build/release/bin/postgresql.jar:/tmp/dist-test-taskWZOx2_/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.131.148:44359/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-06-21 05:12:16,266 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-06-21 05:12:16,266 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWZOx2_/build/release/bin/postgresql.jar:/tmp/dist-test-taskWZOx2_/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.131.148:44359/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-06-21 05:12:16,523 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWZOx2_/build/release/bin/postgresql.jar:/tmp/dist-test-taskWZOx2_/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.131.148:44359/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-06-21 05:12:16,774 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWZOx2_/build/release/bin/postgresql.jar:/tmp/dist-test-taskWZOx2_/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.131.148:44359/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-vrq1.c.gcp-upstream.internal';"
2026-06-21 05:12:17,035 [I] CORE_DB_SCHEMA import status has been updated
2026-06-21 05:12:17,035 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWZOx2_/build/release/bin/postgresql.jar:/tmp/dist-test-taskWZOx2_/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.131.148:44359/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-06-21 05:12:17,291 [I] DB_PATCHES have already been applied
I20260621 05:12:17.299228 3598 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782018616648578-3598-0/minicluster-data/ranger-admin:/tmp/dist-test-taskWZOx2_/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskWZOx2_/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskWZOx2_/build/release/bin/postgresql.jar:/tmp/dist-test-taskWZOx2_/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260621 05:12:17.299319 3598 mini_ranger.cc:193] Using host: 127.3.131.148
I20260621 05:12:17.302984 3598 mini_ranger.cc:227] Ranger admin URL: http://127.3.131.148:42361
Jun 21, 2026 5:12:17 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jun 21, 2026 5:12:17 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jun 21, 2026 5:12:17 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782018616648578-3598-0/minicluster-data/ranger-admin/ews/webapp
Jun 21, 2026 5:12:17 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782018616648578-3598-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Jun 21, 2026 5:12:17 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782018616648578-3598-0/minicluster-data/ranger-admin/ews/webapp] .....
Jun 21, 2026 5:12:18 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782018616648578-3598-0/minicluster-data/ranger-admin/ews/webapp].
Jun 21, 2026 5:12:18 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260621 05:12:18.444518 3598 mini_ranger.cc:160] Time spent starting Ranger: real 9.010s user 0.000s sys 0.010s
Jun 21, 2026 5:12:40 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Jun 21, 2026 5:12:40 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Jun 21, 2026 5:12:40 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Jun 21, 2026 5:12:40 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Jun 21, 2026 5:12:40 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Jun 21, 2026 5:12:40 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Jun 21, 2026 5:12:40 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Jun 21, 2026 5:12:40 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Jun 21, 2026 5:12:40 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Jun 21, 2026 5:12:40 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Jun 21, 2026 5:12:40 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Jun 21, 2026 5:12:40 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Jun 21, 2026 5:12:40 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Jun 21, 2026 5:12:40 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Jun 21, 2026 5:12:40 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Jun 21, 2026 5:12:40 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Jun 21, 2026 5:12:40 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Jun 21, 2026 5:12:41 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260621 05:12:42.114286 3598 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.3.131.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.3.131.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.3.131.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782018616648578-3598-0/krb5kdc/rangerkms_127.3.131.148@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.3.131.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782018616648578-3598-0/krb5kdc/rangerkms_127.3.131.148@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.3.131.148@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.3.131.148@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.3.131.148@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782018616648578-3598-0/krb5kdc/HTTP_127.3.131.148@KRBTEST.COM.keytab.
Entry for principal HTTP/127.3.131.148@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782018616648578-3598-0/krb5kdc/HTTP_127.3.131.148@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Jun 21 05:12:42 dist-test-slave-vrq1 krb5kdc[15364](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782018762, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260621 05:12:42.218202 3598 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260621 05:12:42.229971 3598 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260621 05:12:43.308244 3598 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260621 05:12:43.308322 3598 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782018616648578-3598-0/minicluster-data/ranger-kms
I20260621 05:12:43.308339 3598 mini_ranger_kms.cc:227] Using postgres at 127.3.131.148:44359
2026-06-21 05:12:43,487 [I] DB FLAVOR :POSTGRES
2026-06-21 05:12:43,488 [I] --------- Verifying Ranger DB connection ---------
2026-06-21 05:12:43,488 [I] Checking connection
2026-06-21 05:12:43,488 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWZOx2_/build/release/bin/postgresql.jar:/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782018616648578-3598-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.131.148:44359/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-06-21 05:12:43,751 [I] connection success
2026-06-21 05:12:43,751 [I] --------- Verifying Ranger DB tables ---------
2026-06-21 05:12:43,751 [I] Verifying table ranger_masterkey in database rangerkms
2026-06-21 05:12:43,751 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWZOx2_/build/release/bin/postgresql.jar:/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782018616648578-3598-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.131.148:44359/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-06-21 05:12:44,013 [I] Table ranger_masterkey does not exist in database rangerkms
2026-06-21 05:12:44,014 [I] --------- Importing Ranger Core DB Schema ---------
2026-06-21 05:12:44,014 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-06-21 05:12:44,014 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWZOx2_/build/release/bin/postgresql.jar:/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782018616648578-3598-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.131.148:44359/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782018616648578-3598-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-06-21 05:12:44,332 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260621 05:12:44.858042 3598 mini_ranger_kms.cc:310] Created kms service
I20260621 05:12:45.077405 3598 mini_ranger_kms.cc:326] Created kudu user
I20260621 05:12:45.122552 3598 mini_ranger_kms.cc:343] Created rangerkms user
Jun 21 05:12:45 dist-test-slave-vrq1 krb5kdc[15364](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782018762, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.3.131.148@KRBTEST.COM
I20260621 05:12:45.531028 3598 mini_ranger_kms.cc:384] Added ranger policy
I20260621 05:12:45.531309 3598 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782018616648578-3598-0/minicluster-data/ranger-kms:/tmp/dist-test-taskWZOx2_/build/release/bin/postgresql.jar:/tmp/dist-test-taskWZOx2_/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskWZOx2_/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260621 05:12:45.531347 3598 mini_ranger_kms.cc:240] Using host: 127.3.131.148
I20260621 05:12:45.535620 3598 mini_ranger_kms.cc:276] Ranger KMS PID: 15758
I20260621 05:12:45.535710 3598 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.3.131.148:50409
Jun 21, 2026 5:12:46 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jun 21, 2026 5:12:46 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jun 21, 2026 5:12:46 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782018616648578-3598-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Jun 21, 2026 5:12:46 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782018616648578-3598-0/minicluster-data/ranger-kms/ews/webapp] .....
Jun 21, 2026 5:12:47 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782018616648578-3598-0/minicluster-data/ranger-kms/ews/webapp].
Jun 21, 2026 5:12:47 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
Jun 21 05:12:47 dist-test-slave-vrq1 krb5kdc[15364](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1782018767, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.3.131.148@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 05:12:47.812085 3598 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.594s user 0.000s sys 0.155s
I20260621 05:12:47.812201 3598 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260621 05:12:47.812253 3598 mini_ranger_kms.cc:401] 127.3.131.148:50409/kms/v1/keys
Jun 21 05:12:48 dist-test-slave-vrq1 krb5kdc[15364](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782018767, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.3.131.148@KRBTEST.COM for HTTP/127.3.131.148@KRBTEST.COM
Jun 21 05:12:48 dist-test-slave-vrq1 krb5kdc[15364](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782018767, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.3.131.148@KRBTEST.COM for HTTP/127.3.131.148@KRBTEST.COM
Jun 21 05:12:51 dist-test-slave-vrq1 krb5kdc[15364](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782018771, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.3.131.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.131.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.131.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782018616648578-3598-0/krb5kdc/kudu_127.3.131.190.keytab.
Entry for principal kudu/127.3.131.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782018616648578-3598-0/krb5kdc/kudu_127.3.131.190.keytab.
I20260621 05:12:51.059022 3598 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWZOx2_/build/release/bin/kudu
/tmp/dist-test-taskWZOx2_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782018616648578-3598-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782018616648578-3598-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782018616648578-3598-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782018616648578-3598-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.131.190:34691
--webserver_interface=127.3.131.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782018616648578-3598-0/krb5kdc/kudu_127.3.131.190.keytab
--principal=kudu/127.3.131.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.131.148:45931
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.131.190:34691
--ranger_config_path=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782018616648578-3598-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.3.131.148:50409/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782018616648578-3598-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782018616648578-3598-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782018616648578-3598-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 05:12:51.141503 15808 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 05:12:51.141757 15808 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 05:12:51.141783 15808 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 05:12:51.143074 15808 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 05:12:51.143137 15808 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 05:12:51.143159 15808 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 05:12:51.143194 15808 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 05:12:51.143208 15808 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 05:12:51.145141 15808 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.131.148:45931
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782018616648578-3598-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782018616648578-3598-0/minicluster-data/master-0/wal
--ranger_kms_url=127.3.131.148:50409/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.3.131.190:34691
--ranger_config_path=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782018616648578-3598-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782018616648578-3598-0/krb5kdc/kudu_127.3.131.190.keytab
--principal=kudu/127.3.131.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.131.190:34691
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782018616648578-3598-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.131.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15808
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782018616648578-3598-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 21 Jun 2026 04:16:16 UTC on ccycloud.quasar-xycaha.root.comops.site
build id 4457881
I20260621 05:12:51.145434 15808 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 05:12:51.145709 15808 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260621 05:12:51.151418 15808 server_base.cc:1034] running on GCE node
W20260621 05:12:51.151660 15816 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:12:51.151620 15814 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:12:51.152243 15813 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:12:51.152719 15808 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 05:12:51.153100 15808 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 05:12:51.154271 15808 hybrid_clock.cc:648] HybridClock initialized: now 1782018771154240 us; error 55 us; skew 500 ppm
Jun 21 05:12:51 dist-test-slave-vrq1 krb5kdc[15364](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782018771, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.131.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 05:12:51.166762 15808 init.cc:377] Logged in from keytab as kudu/127.3.131.190@KRBTEST.COM (short username kudu)
I20260621 05:12:51.167527 15808 webserver.cc:460] Webserver started at http://127.3.131.190:36935/ using document root <none> and password file <none>
I20260621 05:12:51.167969 15808 fs_manager.cc:359] Metadata directory not provided
I20260621 05:12:51.168071 15808 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 05:12:51.168242 15808 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 21 05:12:51 dist-test-slave-vrq1 krb5kdc[15364](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782018771, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.131.190@KRBTEST.COM for HTTP/127.3.131.148@KRBTEST.COM
I20260621 05:12:51.233430 15808 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782018616648578-3598-0/minicluster-data/master-0/data/instance:
uuid: "caf4f1ccbc684b5ca79d6a100cec1e37"
format_stamp: "Formatted at 2026-06-21 05:12:51 on dist-test-slave-vrq1"
server_key: "f98f34b974cfc807135df0d928a2e762"
server_key_iv: "6722e13e2ae298ac556c709bee706cfa"
server_key_version: "kuduclusterkey@0"
I20260621 05:12:51.233990 15808 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782018616648578-3598-0/minicluster-data/master-0/wal/instance:
uuid: "caf4f1ccbc684b5ca79d6a100cec1e37"
format_stamp: "Formatted at 2026-06-21 05:12:51 on dist-test-slave-vrq1"
server_key: "f98f34b974cfc807135df0d928a2e762"
server_key_iv: "6722e13e2ae298ac556c709bee706cfa"
server_key_version: "kuduclusterkey@0"
I20260621 05:12:51.236122 15808 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.004s sys 0.000s
I20260621 05:12:51.253427 15823 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 05:12:51.253679 15808 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260621 05:12:51.253790 15808 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782018616648578-3598-0/minicluster-data/master-0/data,/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782018616648578-3598-0/minicluster-data/master-0/wal
uuid: "caf4f1ccbc684b5ca79d6a100cec1e37"
format_stamp: "Formatted at 2026-06-21 05:12:51 on dist-test-slave-vrq1"
server_key: "f98f34b974cfc807135df0d928a2e762"
server_key_iv: "6722e13e2ae298ac556c709bee706cfa"
server_key_version: "kuduclusterkey@0"
I20260621 05:12:51.253902 15808 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782018616648578-3598-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782018616648578-3598-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782018616648578-3598-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 05:12:51.275414 15808 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 05:12:51.277910 15808 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 05:12:51.278076 15808 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 05:12:51.283504 15808 rpc_server.cc:307] RPC server started. Bound to: 127.3.131.190:34691
I20260621 05:12:51.283576 15875 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.131.190:34691 every 8 connection(s)
I20260621 05:12:51.284019 15808 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782018616648578-3598-0/minicluster-data/master-0/data/info.pb
I20260621 05:12:51.284817 15876 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 05:12:51.286165 3598 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWZOx2_/build/release/bin/kudu as pid 15808
I20260621 05:12:51.286283 3598 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782018616648578-3598-0/minicluster-data/master-0/wal/instance
I20260621 05:12:51.286475 3598 external_mini_cluster.cc:1428] Setting key d3a51e935ee5e22d3977daf30288cd48
I20260621 05:12:51.287814 15876 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P caf4f1ccbc684b5ca79d6a100cec1e37: Bootstrap starting.
I20260621 05:12:51.288604 15876 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P caf4f1ccbc684b5ca79d6a100cec1e37: Neither blocks nor log segments found. Creating new log.
I20260621 05:12:51.288980 15876 log.cc:824] T 00000000000000000000000000000000 P caf4f1ccbc684b5ca79d6a100cec1e37: Log is configured to *not* fsync() on all Append() calls
I20260621 05:12:51.289777 15876 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P caf4f1ccbc684b5ca79d6a100cec1e37: No bootstrap required, opened a new log
I20260621 05:12:51.292220 15876 raft_consensus.cc:348] T 00000000000000000000000000000000 P caf4f1ccbc684b5ca79d6a100cec1e37 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "caf4f1ccbc684b5ca79d6a100cec1e37" member_type: VOTER last_known_addr { host: "127.3.131.190" port: 34691 } }
I20260621 05:12:51.292392 15876 raft_consensus.cc:374] T 00000000000000000000000000000000 P caf4f1ccbc684b5ca79d6a100cec1e37 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 05:12:51.292459 15876 raft_consensus.cc:729] T 00000000000000000000000000000000 P caf4f1ccbc684b5ca79d6a100cec1e37 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: caf4f1ccbc684b5ca79d6a100cec1e37, State: Initialized, Role: FOLLOWER
I20260621 05:12:51.292610 15876 consensus_queue.cc:260] T 00000000000000000000000000000000 P caf4f1ccbc684b5ca79d6a100cec1e37 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "caf4f1ccbc684b5ca79d6a100cec1e37" member_type: VOTER last_known_addr { host: "127.3.131.190" port: 34691 } }
I20260621 05:12:51.292711 15876 raft_consensus.cc:388] T 00000000000000000000000000000000 P caf4f1ccbc684b5ca79d6a100cec1e37 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260621 05:12:51.292785 15876 raft_consensus.cc:482] T 00000000000000000000000000000000 P caf4f1ccbc684b5ca79d6a100cec1e37 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260621 05:12:51.292855 15876 raft_consensus.cc:3037] T 00000000000000000000000000000000 P caf4f1ccbc684b5ca79d6a100cec1e37 [term 0 FOLLOWER]: Advancing to term 1
I20260621 05:12:51.293550 15876 raft_consensus.cc:504] T 00000000000000000000000000000000 P caf4f1ccbc684b5ca79d6a100cec1e37 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "caf4f1ccbc684b5ca79d6a100cec1e37" member_type: VOTER last_known_addr { host: "127.3.131.190" port: 34691 } }
I20260621 05:12:51.293673 15876 leader_election.cc:302] T 00000000000000000000000000000000 P caf4f1ccbc684b5ca79d6a100cec1e37 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: caf4f1ccbc684b5ca79d6a100cec1e37; no voters:
I20260621 05:12:51.293970 15876 leader_election.cc:288] T 00000000000000000000000000000000 P caf4f1ccbc684b5ca79d6a100cec1e37 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260621 05:12:51.294361 15876 sys_catalog.cc:564] T 00000000000000000000000000000000 P caf4f1ccbc684b5ca79d6a100cec1e37 [sys.catalog]: configured and running, proceeding with master startup.
Jun 21 05:12:51 dist-test-slave-vrq1 krb5kdc[15364](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782018771, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.131.190@KRBTEST.COM
I20260621 05:12:51.296082 15876 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782018616648578-3598-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260621 05:12:51.296624 15882 raft_consensus.cc:2781] T 00000000000000000000000000000000 P caf4f1ccbc684b5ca79d6a100cec1e37 [term 1 FOLLOWER]: Leader election won for term 1
I20260621 05:12:51.298867 15882 raft_consensus.cc:686] T 00000000000000000000000000000000 P caf4f1ccbc684b5ca79d6a100cec1e37 [term 1 LEADER]: Becoming Leader. State: Replica: caf4f1ccbc684b5ca79d6a100cec1e37, State: Running, Role: LEADER
I20260621 05:12:51.299494 15882 consensus_queue.cc:237] T 00000000000000000000000000000000 P caf4f1ccbc684b5ca79d6a100cec1e37 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "caf4f1ccbc684b5ca79d6a100cec1e37" member_type: VOTER last_known_addr { host: "127.3.131.190" port: 34691 } }
I20260621 05:12:51.300348 15882 sys_catalog.cc:455] T 00000000000000000000000000000000 P caf4f1ccbc684b5ca79d6a100cec1e37 [sys.catalog]: SysCatalogTable state changed. Reason: New leader caf4f1ccbc684b5ca79d6a100cec1e37. Latest consensus state: current_term: 1 leader_uuid: "caf4f1ccbc684b5ca79d6a100cec1e37" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "caf4f1ccbc684b5ca79d6a100cec1e37" member_type: VOTER last_known_addr { host: "127.3.131.190" port: 34691 } } }
I20260621 05:12:51.300472 15882 sys_catalog.cc:458] T 00000000000000000000000000000000 P caf4f1ccbc684b5ca79d6a100cec1e37 [sys.catalog]: This master's current role is: LEADER
I20260621 05:12:51.300951 15897 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260621 05:12:51.301772 15897 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260621 05:12:51.303213 15881 sys_catalog.cc:455] T 00000000000000000000000000000000 P caf4f1ccbc684b5ca79d6a100cec1e37 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "caf4f1ccbc684b5ca79d6a100cec1e37" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "caf4f1ccbc684b5ca79d6a100cec1e37" member_type: VOTER last_known_addr { host: "127.3.131.190" port: 34691 } } }
I20260621 05:12:51.303650 15881 sys_catalog.cc:458] T 00000000000000000000000000000000 P caf4f1ccbc684b5ca79d6a100cec1e37 [sys.catalog]: This master's current role is: LEADER
I20260621 05:12:51.304320 15879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:12:51.287053 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40176 (local address 127.3.131.190:34691)
0621 05:12:51.287284 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0621 05:12:51.287294 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0621 05:12:51.287387 (+ 93us) server_negotiation.cc:407] Connection header received
0621 05:12:51.287983 (+ 596us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:12:51.287994 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:12:51.288062 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:12:51.288221 (+ 159us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:12:51.289097 (+ 876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:12:51.290551 (+ 1454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:12:51.291705 (+ 1154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:12:51.292357 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:12:51.292827 (+ 470us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:12:51.295553 (+ 2726us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:12:51.295559 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:12:51.295559 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:12:51.295573 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:12:51.301072 (+ 5499us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:12:51.301740 (+ 668us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:12:51.301744 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:12:51.301745 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:12:51.301934 (+ 189us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:12:51.302197 (+ 263us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:12:51.302197 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:12:51.302198 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:12:51.302797 (+ 599us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:12:51.303491 (+ 694us) server_negotiation.cc:1035] Waiting for connection context
0621 05:12:51.303575 (+ 84us) server_negotiation.cc:299] Negotiation successful
0621 05:12:51.304018 (+ 443us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":158,"thread_start_us":80,"threads_started":1}
I20260621 05:12:51.307904 15897 catalog_manager.cc:1347] Generated new cluster ID: 2bb938f402e54c78aef2f408b7fe7ab5
I20260621 05:12:51.307966 15897 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260621 05:12:51.318753 15897 catalog_manager.cc:1370] Generated new certificate authority record
I20260621 05:12:51.319597 15897 catalog_manager.cc:1504] Loading token signing keys...
I20260621 05:12:51.328444 15897 catalog_manager.cc:5868] T 00000000000000000000000000000000 P caf4f1ccbc684b5ca79d6a100cec1e37: Generated new TSK 0
I20260621 05:12:51.328689 15897 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.3.131.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.131.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.131.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782018616648578-3598-0/krb5kdc/kudu_127.3.131.129.keytab.
Entry for principal kudu/127.3.131.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782018616648578-3598-0/krb5kdc/kudu_127.3.131.129.keytab.
I20260621 05:12:52.040140 3598 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWZOx2_/build/release/bin/kudu
/tmp/dist-test-taskWZOx2_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782018616648578-3598-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782018616648578-3598-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782018616648578-3598-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782018616648578-3598-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.3.131.148:50409/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.131.129:0
--local_ip_for_outbound_sockets=127.3.131.129
--webserver_interface=127.3.131.129
--webserver_port=0
--tserver_master_addrs=127.3.131.190:34691
--keytab_file=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782018616648578-3598-0/krb5kdc/kudu_127.3.131.129.keytab
--principal=kudu/127.3.131.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.131.148:45931
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782018616648578-3598-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782018616648578-3598-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782018616648578-3598-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 05:12:52.136307 15921 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 05:12:52.136610 15921 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 05:12:52.136713 15921 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 05:12:52.138204 15921 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 05:12:52.138288 15921 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 05:12:52.138336 15921 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.131.129
I20260621 05:12:52.140319 15921 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.131.148:45931
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782018616648578-3598-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782018616648578-3598-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.3.131.148:50409/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782018616648578-3598-0/krb5kdc/kudu_127.3.131.129.keytab
--principal=kudu/127.3.131.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.131.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782018616648578-3598-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.131.129
--webserver_port=0
--tserver_master_addrs=127.3.131.190:34691
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15921
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.131.129
--log_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782018616648578-3598-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 21 Jun 2026 04:16:16 UTC on ccycloud.quasar-xycaha.root.comops.site
build id 4457881
I20260621 05:12:52.140640 15921 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 05:12:52.140929 15921 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 05:12:52.146921 15926 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:12:52.146912 15929 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:12:52.147409 15927 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:12:52.148124 15921 server_base.cc:1034] running on GCE node
I20260621 05:12:52.148396 15921 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 05:12:52.148973 15921 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 05:12:52.150187 15921 hybrid_clock.cc:648] HybridClock initialized: now 1782018772150150 us; error 42 us; skew 500 ppm
Jun 21 05:12:52 dist-test-slave-vrq1 krb5kdc[15364](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782018772, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.131.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 05:12:52.153270 15921 init.cc:377] Logged in from keytab as kudu/127.3.131.129@KRBTEST.COM (short username kudu)
I20260621 05:12:52.153759 15921 webserver.cc:460] Webserver started at http://127.3.131.129:35389/ using document root <none> and password file <none>
I20260621 05:12:52.154009 15921 fs_manager.cc:359] Metadata directory not provided
I20260621 05:12:52.154078 15921 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 05:12:52.154222 15921 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 21 05:12:52 dist-test-slave-vrq1 krb5kdc[15364](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782018772, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.131.129@KRBTEST.COM for HTTP/127.3.131.148@KRBTEST.COM
I20260621 05:12:52.170009 15921 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782018616648578-3598-0/minicluster-data/ts-0/data/instance:
uuid: "cddc12c2e002400ea87d54d68d4ceec6"
format_stamp: "Formatted at 2026-06-21 05:12:52 on dist-test-slave-vrq1"
server_key: "73b15661824e25ba58aca5d5b03ba788"
server_key_iv: "7ed46e59df40f18a389be04a660a6722"
server_key_version: "kuduclusterkey@0"
I20260621 05:12:52.170464 15921 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782018616648578-3598-0/minicluster-data/ts-0/wal/instance:
uuid: "cddc12c2e002400ea87d54d68d4ceec6"
format_stamp: "Formatted at 2026-06-21 05:12:52 on dist-test-slave-vrq1"
server_key: "73b15661824e25ba58aca5d5b03ba788"
server_key_iv: "7ed46e59df40f18a389be04a660a6722"
server_key_version: "kuduclusterkey@0"
I20260621 05:12:52.175824 15921 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.000s
I20260621 05:12:52.188851 15936 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 05:12:52.189062 15921 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260621 05:12:52.189159 15921 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782018616648578-3598-0/minicluster-data/ts-0/data,/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782018616648578-3598-0/minicluster-data/ts-0/wal
uuid: "cddc12c2e002400ea87d54d68d4ceec6"
format_stamp: "Formatted at 2026-06-21 05:12:52 on dist-test-slave-vrq1"
server_key: "73b15661824e25ba58aca5d5b03ba788"
server_key_iv: "7ed46e59df40f18a389be04a660a6722"
server_key_version: "kuduclusterkey@0"
I20260621 05:12:52.189275 15921 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782018616648578-3598-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782018616648578-3598-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782018616648578-3598-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 05:12:52.212360 15921 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 05:12:52.214943 15921 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 05:12:52.215119 15921 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 05:12:52.215487 15921 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 05:12:52.215876 15921 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 05:12:52.215929 15921 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 05:12:52.215973 15921 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 05:12:52.216001 15921 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 05:12:52.223879 15921 rpc_server.cc:307] RPC server started. Bound to: 127.3.131.129:43057
I20260621 05:12:52.223915 16049 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.131.129:43057 every 8 connection(s)
I20260621 05:12:52.224386 15921 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782018616648578-3598-0/minicluster-data/ts-0/data/info.pb
I20260621 05:12:52.227069 3598 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWZOx2_/build/release/bin/kudu as pid 15921
I20260621 05:12:52.227209 3598 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782018616648578-3598-0/minicluster-data/ts-0/wal/instance
I20260621 05:12:52.227398 3598 external_mini_cluster.cc:1428] Setting key 599b7c4ba8640f9072868fff9a118da2
Jun 21 05:12:52 dist-test-slave-vrq1 krb5kdc[15364](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782018772, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.131.129@KRBTEST.COM for kudu/127.3.131.190@KRBTEST.COM
I20260621 05:12:52.237990 16053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:12:52.225340 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.131.129:41219 (local address 127.3.131.190:34691)
0621 05:12:52.225481 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0621 05:12:52.225484 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0621 05:12:52.225652 (+ 168us) server_negotiation.cc:407] Connection header received
0621 05:12:52.226303 (+ 651us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:12:52.226305 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:12:52.226320 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:12:52.226402 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:12:52.227246 (+ 844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:12:52.228027 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:12:52.229570 (+ 1543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:12:52.229877 (+ 307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:12:52.229956 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:12:52.233622 (+ 3666us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:12:52.233625 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:12:52.233625 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:12:52.233636 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:12:52.235940 (+ 2304us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:12:52.236645 (+ 705us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:12:52.236648 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:12:52.236648 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:12:52.236731 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:12:52.237114 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:12:52.237115 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:12:52.237116 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:12:52.237297 (+ 181us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:12:52.237368 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0621 05:12:52.237786 (+ 418us) server_negotiation.cc:299] Negotiation successful
0621 05:12:52.237885 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":116,"thread_start_us":79,"threads_started":1}
I20260621 05:12:52.238183 16052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:12:52.225259 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.131.190:34691 (local address 127.3.131.129:41219)
0621 05:12:52.225482 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0621 05:12:52.225535 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0621 05:12:52.226161 (+ 626us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:12:52.226500 (+ 339us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:12:52.226505 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:12:52.226787 (+ 282us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:12:52.227083 (+ 296us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:12:52.227090 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:12:52.228151 (+ 1061us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:12:52.228152 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:12:52.229458 (+ 1306us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:12:52.229462 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:12:52.230014 (+ 552us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:12:52.230015 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:12:52.230203 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:12:52.231894 (+ 1691us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 05:12:52.231906 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:12:52.233455 (+ 1549us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 05:12:52.236048 (+ 2593us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:12:52.236061 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:12:52.236064 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:12:52.236506 (+ 442us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:12:52.236836 (+ 330us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:12:52.236837 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:12:52.236837 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:12:52.237021 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:12:52.237401 (+ 380us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 05:12:52.237408 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 05:12:52.237608 (+ 200us) client_negotiation.cc:769] Sending connection context
0621 05:12:52.237741 (+ 133us) client_negotiation.cc:240] Negotiation successful
0621 05:12:52.237915 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":173,"thread_start_us":107,"threads_started":1}
I20260621 05:12:52.238816 16050 heartbeater.cc:347] Connected to a master server at 127.3.131.190:34691
I20260621 05:12:52.238941 16050 heartbeater.cc:464] Registering TS with master...
I20260621 05:12:52.239140 16050 heartbeater.cc:511] Master 127.3.131.190:34691 requested a full tablet report, sending...
I20260621 05:12:52.239720 15839 ts_manager.cc:194] Registered new tserver with Master: cddc12c2e002400ea87d54d68d4ceec6 (127.3.131.129:43057)
WARNING: no policy specified for kudu/127.3.131.130@KRBTEST.COM; defaulting to no policy
I20260621 05:12:52.240549 15839 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.131.129@KRBTEST.COM'} at 127.3.131.129:41219
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.131.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.131.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782018616648578-3598-0/krb5kdc/kudu_127.3.131.130.keytab.
Entry for principal kudu/127.3.131.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782018616648578-3598-0/krb5kdc/kudu_127.3.131.130.keytab.
I20260621 05:12:52.258757 3598 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWZOx2_/build/release/bin/kudu
/tmp/dist-test-taskWZOx2_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782018616648578-3598-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782018616648578-3598-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782018616648578-3598-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782018616648578-3598-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.3.131.148:50409/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.131.130:0
--local_ip_for_outbound_sockets=127.3.131.130
--webserver_interface=127.3.131.130
--webserver_port=0
--tserver_master_addrs=127.3.131.190:34691
--keytab_file=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782018616648578-3598-0/krb5kdc/kudu_127.3.131.130.keytab
--principal=kudu/127.3.131.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.131.148:45931
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782018616648578-3598-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782018616648578-3598-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782018616648578-3598-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 05:12:52.364010 16058 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 05:12:52.364389 16058 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 05:12:52.364487 16058 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 05:12:52.366386 16058 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 05:12:52.366514 16058 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 05:12:52.366609 16058 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.131.130
I20260621 05:12:52.368460 16058 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.131.148:45931
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782018616648578-3598-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782018616648578-3598-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.3.131.148:50409/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782018616648578-3598-0/krb5kdc/kudu_127.3.131.130.keytab
--principal=kudu/127.3.131.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.131.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782018616648578-3598-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.131.130
--webserver_port=0
--tserver_master_addrs=127.3.131.190:34691
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16058
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.131.130
--log_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782018616648578-3598-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 21 Jun 2026 04:16:16 UTC on ccycloud.quasar-xycaha.root.comops.site
build id 4457881
I20260621 05:12:52.368760 16058 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 05:12:52.369025 16058 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 05:12:52.374578 16066 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:12:52.374701 16065 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:12:52.374606 16068 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:12:52.375448 16058 server_base.cc:1034] running on GCE node
I20260621 05:12:52.375712 16058 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 05:12:52.376119 16058 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 05:12:52.377315 16058 hybrid_clock.cc:648] HybridClock initialized: now 1782018772377294 us; error 36 us; skew 500 ppm
Jun 21 05:12:52 dist-test-slave-vrq1 krb5kdc[15364](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782018772, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.131.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 05:12:52.380143 16058 init.cc:377] Logged in from keytab as kudu/127.3.131.130@KRBTEST.COM (short username kudu)
I20260621 05:12:52.380656 16058 webserver.cc:460] Webserver started at http://127.3.131.130:34175/ using document root <none> and password file <none>
I20260621 05:12:52.380923 16058 fs_manager.cc:359] Metadata directory not provided
I20260621 05:12:52.380993 16058 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 05:12:52.381130 16058 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 21 05:12:52 dist-test-slave-vrq1 krb5kdc[15364](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782018772, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.131.130@KRBTEST.COM for HTTP/127.3.131.148@KRBTEST.COM
I20260621 05:12:52.397344 16058 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782018616648578-3598-0/minicluster-data/ts-1/data/instance:
uuid: "a3fbf94490ec4c3d8e35edf5aaaef40a"
format_stamp: "Formatted at 2026-06-21 05:12:52 on dist-test-slave-vrq1"
server_key: "1fd2eccf9b69f8673872a9ddad22a23f"
server_key_iv: "7b13a0fc7f016c7f9e602db7be5b4821"
server_key_version: "kuduclusterkey@0"
I20260621 05:12:52.397743 16058 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782018616648578-3598-0/minicluster-data/ts-1/wal/instance:
uuid: "a3fbf94490ec4c3d8e35edf5aaaef40a"
format_stamp: "Formatted at 2026-06-21 05:12:52 on dist-test-slave-vrq1"
server_key: "1fd2eccf9b69f8673872a9ddad22a23f"
server_key_iv: "7b13a0fc7f016c7f9e602db7be5b4821"
server_key_version: "kuduclusterkey@0"
I20260621 05:12:52.399227 16058 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260621 05:12:52.423307 16075 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 05:12:52.423645 16058 fs_manager.cc:725] Time spent opening block manager: real 0.011s user 0.001s sys 0.000s
I20260621 05:12:52.423791 16058 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782018616648578-3598-0/minicluster-data/ts-1/data,/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782018616648578-3598-0/minicluster-data/ts-1/wal
uuid: "a3fbf94490ec4c3d8e35edf5aaaef40a"
format_stamp: "Formatted at 2026-06-21 05:12:52 on dist-test-slave-vrq1"
server_key: "1fd2eccf9b69f8673872a9ddad22a23f"
server_key_iv: "7b13a0fc7f016c7f9e602db7be5b4821"
server_key_version: "kuduclusterkey@0"
I20260621 05:12:52.423967 16058 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782018616648578-3598-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782018616648578-3598-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782018616648578-3598-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
Jun 21 05:12:52 dist-test-slave-vrq1 krb5kdc[15364](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1782018772, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.131.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 05:12:52.436053 16058 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 05:12:52.438558 16058 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 05:12:52.438719 16058 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 05:12:52.438977 16058 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 05:12:52.439416 16058 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 05:12:52.439464 16058 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 05:12:52.439495 16058 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 05:12:52.439513 16058 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 05:12:52.446013 16058 rpc_server.cc:307] RPC server started. Bound to: 127.3.131.130:45271
I20260621 05:12:52.446485 16058 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782018616648578-3598-0/minicluster-data/ts-1/data/info.pb
I20260621 05:12:52.448361 3598 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWZOx2_/build/release/bin/kudu as pid 16058
I20260621 05:12:52.448462 3598 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782018616648578-3598-0/minicluster-data/ts-1/wal/instance
I20260621 05:12:52.448665 3598 external_mini_cluster.cc:1428] Setting key 35f8c6e5b143d24d125883f787088815
I20260621 05:12:52.449613 16188 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.131.130:45271 every 8 connection(s)
WARNING: no policy specified for kudu/127.3.131.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.131.131@KRBTEST.COM" created.
Jun 21 05:12:52 dist-test-slave-vrq1 krb5kdc[15364](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782018772, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.131.130@KRBTEST.COM for kudu/127.3.131.190@KRBTEST.COM
I20260621 05:12:52.473958 16192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:12:52.456172 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.131.190:34691 (local address 127.3.131.130:38185)
0621 05:12:52.456414 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0621 05:12:52.456489 (+ 75us) client_negotiation.cc:174] Beginning negotiation
0621 05:12:52.457187 (+ 698us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:12:52.457629 (+ 442us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:12:52.457633 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:12:52.457912 (+ 279us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:12:52.458206 (+ 294us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:12:52.458212 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:12:52.461472 (+ 3260us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:12:52.461473 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:12:52.466191 (+ 4718us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:12:52.466197 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:12:52.466964 (+ 767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:12:52.466966 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:12:52.467143 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:12:52.467858 (+ 715us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 05:12:52.467869 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:12:52.470131 (+ 2262us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 05:12:52.472257 (+ 2126us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:12:52.472266 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:12:52.472268 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:12:52.472584 (+ 316us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:12:52.472805 (+ 221us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:12:52.472806 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:12:52.472806 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:12:52.472943 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:12:52.473244 (+ 301us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 05:12:52.473250 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 05:12:52.473419 (+ 169us) client_negotiation.cc:769] Sending connection context
0621 05:12:52.473523 (+ 104us) client_negotiation.cc:240] Negotiation successful
0621 05:12:52.473687 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":180,"thread_start_us":124,"threads_started":1}
I20260621 05:12:52.474566 16053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:12:52.456191 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.131.130:38185 (local address 127.3.131.190:34691)
0621 05:12:52.456253 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0621 05:12:52.456255 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:12:52.456584 (+ 329us) server_negotiation.cc:407] Connection header received
0621 05:12:52.457372 (+ 788us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:12:52.457374 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:12:52.457391 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:12:52.457422 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:12:52.458344 (+ 922us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:12:52.459114 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:12:52.466352 (+ 7238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:12:52.466697 (+ 345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:12:52.466934 (+ 237us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:12:52.470299 (+ 3365us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:12:52.470301 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:12:52.470301 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:12:52.470311 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:12:52.472116 (+ 1805us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:12:52.472675 (+ 559us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:12:52.472676 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:12:52.472677 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:12:52.472738 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:12:52.473008 (+ 270us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:12:52.473009 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:12:52.473009 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:12:52.473151 (+ 142us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:12:52.473209 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0621 05:12:52.474374 (+ 1165us) server_negotiation.cc:299] Negotiation successful
0621 05:12:52.474475 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260621 05:12:52.474987 16189 heartbeater.cc:347] Connected to a master server at 127.3.131.190:34691
I20260621 05:12:52.475088 16189 heartbeater.cc:464] Registering TS with master...
I20260621 05:12:52.475306 16189 heartbeater.cc:511] Master 127.3.131.190:34691 requested a full tablet report, sending...
I20260621 05:12:52.475694 15839 ts_manager.cc:194] Registered new tserver with Master: a3fbf94490ec4c3d8e35edf5aaaef40a (127.3.131.130:45271)
I20260621 05:12:52.476207 15839 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.131.130@KRBTEST.COM'} at 127.3.131.130:38185
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.131.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782018616648578-3598-0/krb5kdc/kudu_127.3.131.131.keytab.
Entry for principal kudu/127.3.131.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782018616648578-3598-0/krb5kdc/kudu_127.3.131.131.keytab.
I20260621 05:12:52.488571 3598 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWZOx2_/build/release/bin/kudu
/tmp/dist-test-taskWZOx2_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782018616648578-3598-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782018616648578-3598-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782018616648578-3598-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782018616648578-3598-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.3.131.148:50409/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.131.131:0
--local_ip_for_outbound_sockets=127.3.131.131
--webserver_interface=127.3.131.131
--webserver_port=0
--tserver_master_addrs=127.3.131.190:34691
--keytab_file=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782018616648578-3598-0/krb5kdc/kudu_127.3.131.131.keytab
--principal=kudu/127.3.131.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.131.148:45931
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782018616648578-3598-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782018616648578-3598-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782018616648578-3598-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 05:12:52.621098 16194 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 05:12:52.621436 16194 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 05:12:52.621482 16194 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 05:12:52.623612 16194 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 05:12:52.623699 16194 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 05:12:52.623756 16194 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.131.131
I20260621 05:12:52.626335 16194 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.131.148:45931
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782018616648578-3598-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782018616648578-3598-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.3.131.148:50409/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782018616648578-3598-0/krb5kdc/kudu_127.3.131.131.keytab
--principal=kudu/127.3.131.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.131.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782018616648578-3598-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.131.131
--webserver_port=0
--tserver_master_addrs=127.3.131.190:34691
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16194
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.131.131
--log_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782018616648578-3598-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 21 Jun 2026 04:16:16 UTC on ccycloud.quasar-xycaha.root.comops.site
build id 4457881
I20260621 05:12:52.626639 16194 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 05:12:52.626962 16194 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 05:12:52.634109 16201 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:12:52.634141 16203 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:12:52.634536 16200 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:12:52.634694 16194 server_base.cc:1034] running on GCE node
I20260621 05:12:52.635043 16194 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 05:12:52.635428 16194 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 05:12:52.636646 16194 hybrid_clock.cc:648] HybridClock initialized: now 1782018772636622 us; error 38 us; skew 500 ppm
Jun 21 05:12:52 dist-test-slave-vrq1 krb5kdc[15364](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782018772, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.131.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 05:12:52.640316 16194 init.cc:377] Logged in from keytab as kudu/127.3.131.131@KRBTEST.COM (short username kudu)
I20260621 05:12:52.640949 16194 webserver.cc:460] Webserver started at http://127.3.131.131:43025/ using document root <none> and password file <none>
I20260621 05:12:52.641249 16194 fs_manager.cc:359] Metadata directory not provided
I20260621 05:12:52.641328 16194 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 05:12:52.641486 16194 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 21 05:12:52 dist-test-slave-vrq1 krb5kdc[15364](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782018772, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.131.131@KRBTEST.COM for HTTP/127.3.131.148@KRBTEST.COM
I20260621 05:12:52.659945 16194 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782018616648578-3598-0/minicluster-data/ts-2/data/instance:
uuid: "4644e1517d854a16a6716ea38148e104"
format_stamp: "Formatted at 2026-06-21 05:12:52 on dist-test-slave-vrq1"
server_key: "fc74f8844f70f259a73bc0386d6e1ef3"
server_key_iv: "3f44ac5c541a50708d45eef90f9934a9"
server_key_version: "kuduclusterkey@0"
I20260621 05:12:52.660555 16194 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782018616648578-3598-0/minicluster-data/ts-2/wal/instance:
uuid: "4644e1517d854a16a6716ea38148e104"
format_stamp: "Formatted at 2026-06-21 05:12:52 on dist-test-slave-vrq1"
server_key: "fc74f8844f70f259a73bc0386d6e1ef3"
server_key_iv: "3f44ac5c541a50708d45eef90f9934a9"
server_key_version: "kuduclusterkey@0"
I20260621 05:12:52.663684 16194 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.000s
I20260621 05:12:52.679577 16210 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 05:12:52.687682 16194 fs_manager.cc:725] Time spent opening block manager: real 0.010s user 0.001s sys 0.000s
I20260621 05:12:52.687849 16194 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782018616648578-3598-0/minicluster-data/ts-2/data,/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782018616648578-3598-0/minicluster-data/ts-2/wal
uuid: "4644e1517d854a16a6716ea38148e104"
format_stamp: "Formatted at 2026-06-21 05:12:52 on dist-test-slave-vrq1"
server_key: "fc74f8844f70f259a73bc0386d6e1ef3"
server_key_iv: "3f44ac5c541a50708d45eef90f9934a9"
server_key_version: "kuduclusterkey@0"
I20260621 05:12:52.687980 16194 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782018616648578-3598-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782018616648578-3598-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782018616648578-3598-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 05:12:52.705385 16194 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 05:12:52.708110 16194 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 05:12:52.708266 16194 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 05:12:52.708562 16194 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 05:12:52.708958 16194 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 05:12:52.709008 16194 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 05:12:52.709040 16194 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 05:12:52.709069 16194 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 05:12:52.714812 16194 rpc_server.cc:307] RPC server started. Bound to: 127.3.131.131:33035
I20260621 05:12:52.718659 16194 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782018616648578-3598-0/minicluster-data/ts-2/data/info.pb
I20260621 05:12:52.721482 16323 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.131.131:33035 every 8 connection(s)
I20260621 05:12:52.727566 3598 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWZOx2_/build/release/bin/kudu as pid 16194
I20260621 05:12:52.727684 3598 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782018616648578-3598-0/minicluster-data/ts-2/wal/instance
I20260621 05:12:52.727895 3598 external_mini_cluster.cc:1428] Setting key d65ed2ae655ad8738d11ea12474434d9
Jun 21 05:12:52 dist-test-slave-vrq1 krb5kdc[15364](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782018772, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.131.131@KRBTEST.COM for kudu/127.3.131.190@KRBTEST.COM
I20260621 05:12:52.736897 16053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:12:52.722863 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.131.131:57357 (local address 127.3.131.190:34691)
0621 05:12:52.723252 (+ 389us) server_negotiation.cc:206] Beginning negotiation
0621 05:12:52.723254 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:12:52.724638 (+ 1384us) server_negotiation.cc:407] Connection header received
0621 05:12:52.725428 (+ 790us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:12:52.725429 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:12:52.725443 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:12:52.725477 (+ 34us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:12:52.726476 (+ 999us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:12:52.727312 (+ 836us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:12:52.730189 (+ 2877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:12:52.730489 (+ 300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:12:52.730531 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:12:52.733271 (+ 2740us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:12:52.733273 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:12:52.733273 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:12:52.733282 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:12:52.735004 (+ 1722us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:12:52.735669 (+ 665us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:12:52.735670 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:12:52.735671 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:12:52.735742 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:12:52.736080 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:12:52.736081 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:12:52.736081 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:12:52.736232 (+ 151us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:12:52.736302 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0621 05:12:52.736696 (+ 394us) server_negotiation.cc:299] Negotiation successful
0621 05:12:52.736796 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":353}
I20260621 05:12:52.737097 16326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:12:52.722536 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.131.190:34691 (local address 127.3.131.131:57357)
0621 05:12:52.724476 (+ 1940us) negotiation.cc:107] Waiting for socket to connect
0621 05:12:52.724531 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0621 05:12:52.725289 (+ 758us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:12:52.725687 (+ 398us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:12:52.725692 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:12:52.726056 (+ 364us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:12:52.726355 (+ 299us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:12:52.726362 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:12:52.728859 (+ 2497us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:12:52.728860 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:12:52.730080 (+ 1220us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:12:52.730083 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:12:52.730636 (+ 553us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:12:52.730637 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:12:52.730778 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:12:52.731491 (+ 713us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 05:12:52.731503 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:12:52.733149 (+ 1646us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 05:12:52.735120 (+ 1971us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:12:52.735129 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:12:52.735132 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:12:52.735559 (+ 427us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:12:52.735831 (+ 272us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:12:52.735832 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:12:52.735833 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:12:52.735995 (+ 162us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:12:52.736307 (+ 312us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 05:12:52.736313 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 05:12:52.736515 (+ 202us) client_negotiation.cc:769] Sending connection context
0621 05:12:52.736642 (+ 127us) client_negotiation.cc:240] Negotiation successful
0621 05:12:52.736833 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1883,"thread_start_us":104,"threads_started":1}
I20260621 05:12:52.737660 16324 heartbeater.cc:347] Connected to a master server at 127.3.131.190:34691
I20260621 05:12:52.737787 16324 heartbeater.cc:464] Registering TS with master...
I20260621 05:12:52.738003 16324 heartbeater.cc:511] Master 127.3.131.190:34691 requested a full tablet report, sending...
I20260621 05:12:52.738487 15839 ts_manager.cc:194] Registered new tserver with Master: 4644e1517d854a16a6716ea38148e104 (127.3.131.131:33035)
I20260621 05:12:52.739002 15839 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.131.131@KRBTEST.COM'} at 127.3.131.131:57357
I20260621 05:12:52.740516 3598 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260621 05:12:52.741237 3598 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWZOx2_/build/release/bin/kudu with pid 15921
I20260621 05:12:52.748304 3598 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWZOx2_/build/release/bin/kudu with pid 16058
I20260621 05:12:52.754222 3598 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWZOx2_/build/release/bin/kudu with pid 16194
I20260621 05:12:52.760574 3598 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWZOx2_/build/release/bin/kudu with pid 15808
I20260621 05:12:52.767998 3598 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWZOx2_/build/release/bin/kudu
/tmp/dist-test-taskWZOx2_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782018616648578-3598-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782018616648578-3598-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782018616648578-3598-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782018616648578-3598-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.131.190:34691
--webserver_interface=127.3.131.190
--webserver_port=36935
--keytab_file=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782018616648578-3598-0/krb5kdc/kudu_127.3.131.190.keytab
--principal=kudu/127.3.131.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.131.148:45931
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.131.190:34691
--ranger_config_path=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782018616648578-3598-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.3.131.148:50409/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.3.131.148:50409/kms with env {KRB5CCNAME=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782018616648578-3598-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782018616648578-3598-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782018616648578-3598-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 05:12:52.852342 16327 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 05:12:52.852602 16327 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 05:12:52.852643 16327 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 05:12:52.853852 16327 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 05:12:52.853917 16327 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 05:12:52.853940 16327 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 05:12:52.853964 16327 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 05:12:52.853989 16327 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 05:12:52.855734 16327 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.131.148:45931
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782018616648578-3598-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782018616648578-3598-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.3.131.148:50409/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.3.131.190:34691
--ranger_config_path=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782018616648578-3598-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782018616648578-3598-0/krb5kdc/kudu_127.3.131.190.keytab
--principal=kudu/127.3.131.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.131.190:34691
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782018616648578-3598-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.131.190
--webserver_port=36935
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16327
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782018616648578-3598-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 21 Jun 2026 04:16:16 UTC on ccycloud.quasar-xycaha.root.comops.site
build id 4457881
I20260621 05:12:52.856015 16327 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 05:12:52.856243 16327 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 05:12:52.862120 16332 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:12:52.862185 16335 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:12:52.862308 16333 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:12:52.862651 16327 server_base.cc:1034] running on GCE node
I20260621 05:12:52.862998 16327 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 05:12:52.863397 16327 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 05:12:52.864583 16327 hybrid_clock.cc:648] HybridClock initialized: now 1782018772864554 us; error 42 us; skew 500 ppm
Jun 21 05:12:52 dist-test-slave-vrq1 krb5kdc[15364](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782018772, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.131.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 05:12:52.867517 16327 init.cc:377] Logged in from keytab as kudu/127.3.131.190@KRBTEST.COM (short username kudu)
I20260621 05:12:52.868109 16327 webserver.cc:460] Webserver started at http://127.3.131.190:36935/ using document root <none> and password file <none>
I20260621 05:12:52.868378 16327 fs_manager.cc:359] Metadata directory not provided
I20260621 05:12:52.868438 16327 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jun 21 05:12:52 dist-test-slave-vrq1 krb5kdc[15364](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782018772, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.131.190@KRBTEST.COM for HTTP/127.3.131.148@KRBTEST.COM
I20260621 05:12:52.949678 16327 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260621 05:12:52.950625 16343 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 05:12:52.950871 16327 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260621 05:12:52.950976 16327 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782018616648578-3598-0/minicluster-data/master-0/data,/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782018616648578-3598-0/minicluster-data/master-0/wal
uuid: "caf4f1ccbc684b5ca79d6a100cec1e37"
format_stamp: "Formatted at 2026-06-21 05:12:51 on dist-test-slave-vrq1"
server_key: "f98f34b974cfc807135df0d928a2e762"
server_key_iv: "6722e13e2ae298ac556c709bee706cfa"
server_key_version: "kuduclusterkey@0"
I20260621 05:12:52.951496 16327 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782018616648578-3598-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782018616648578-3598-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782018616648578-3598-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 05:12:52.967391 16327 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 05:12:52.969636 16327 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 05:12:52.969785 16327 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 05:12:52.975016 16327 rpc_server.cc:307] RPC server started. Bound to: 127.3.131.190:34691
I20260621 05:12:52.975126 16395 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.131.190:34691 every 8 connection(s)
I20260621 05:12:52.975559 16327 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782018616648578-3598-0/minicluster-data/master-0/data/info.pb
I20260621 05:12:52.976933 16396 sys_catalog.cc:263] Verifying existing consensus state
I20260621 05:12:52.977586 16396 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P caf4f1ccbc684b5ca79d6a100cec1e37: Bootstrap starting.
I20260621 05:12:52.985891 3598 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWZOx2_/build/release/bin/kudu as pid 16327
I20260621 05:12:52.986018 16396 log.cc:824] T 00000000000000000000000000000000 P caf4f1ccbc684b5ca79d6a100cec1e37: Log is configured to *not* fsync() on all Append() calls
I20260621 05:12:52.986414 3598 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWZOx2_/build/release/bin/kudu
/tmp/dist-test-taskWZOx2_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782018616648578-3598-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782018616648578-3598-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782018616648578-3598-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782018616648578-3598-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.3.131.148:50409/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.131.129:43057
--local_ip_for_outbound_sockets=127.3.131.129
--tserver_master_addrs=127.3.131.190:34691
--webserver_port=35389
--webserver_interface=127.3.131.129
--keytab_file=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782018616648578-3598-0/krb5kdc/kudu_127.3.131.129.keytab
--principal=kudu/127.3.131.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.131.148:45931
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.3.131.148:50409/kms with env {KRB5CCNAME=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782018616648578-3598-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782018616648578-3598-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782018616648578-3598-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260621 05:12:52.987979 16396 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P caf4f1ccbc684b5ca79d6a100cec1e37: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260621 05:12:52.988287 16396 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P caf4f1ccbc684b5ca79d6a100cec1e37: Bootstrap complete.
I20260621 05:12:52.990821 16396 raft_consensus.cc:348] T 00000000000000000000000000000000 P caf4f1ccbc684b5ca79d6a100cec1e37 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "caf4f1ccbc684b5ca79d6a100cec1e37" member_type: VOTER last_known_addr { host: "127.3.131.190" port: 34691 } }
I20260621 05:12:52.991097 16396 raft_consensus.cc:729] T 00000000000000000000000000000000 P caf4f1ccbc684b5ca79d6a100cec1e37 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: caf4f1ccbc684b5ca79d6a100cec1e37, State: Initialized, Role: FOLLOWER
I20260621 05:12:52.991341 16396 consensus_queue.cc:260] T 00000000000000000000000000000000 P caf4f1ccbc684b5ca79d6a100cec1e37 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "caf4f1ccbc684b5ca79d6a100cec1e37" member_type: VOTER last_known_addr { host: "127.3.131.190" port: 34691 } }
I20260621 05:12:52.991437 16396 raft_consensus.cc:388] T 00000000000000000000000000000000 P caf4f1ccbc684b5ca79d6a100cec1e37 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260621 05:12:52.991487 16396 raft_consensus.cc:482] T 00000000000000000000000000000000 P caf4f1ccbc684b5ca79d6a100cec1e37 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260621 05:12:52.991535 16396 raft_consensus.cc:3037] T 00000000000000000000000000000000 P caf4f1ccbc684b5ca79d6a100cec1e37 [term 1 FOLLOWER]: Advancing to term 2
I20260621 05:12:52.992661 16396 raft_consensus.cc:504] T 00000000000000000000000000000000 P caf4f1ccbc684b5ca79d6a100cec1e37 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "caf4f1ccbc684b5ca79d6a100cec1e37" member_type: VOTER last_known_addr { host: "127.3.131.190" port: 34691 } }
I20260621 05:12:52.992790 16396 leader_election.cc:302] T 00000000000000000000000000000000 P caf4f1ccbc684b5ca79d6a100cec1e37 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: caf4f1ccbc684b5ca79d6a100cec1e37; no voters:
I20260621 05:12:52.992998 16396 leader_election.cc:288] T 00000000000000000000000000000000 P caf4f1ccbc684b5ca79d6a100cec1e37 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260621 05:12:52.993110 16400 raft_consensus.cc:2781] T 00000000000000000000000000000000 P caf4f1ccbc684b5ca79d6a100cec1e37 [term 2 FOLLOWER]: Leader election won for term 2
I20260621 05:12:52.993290 16400 raft_consensus.cc:686] T 00000000000000000000000000000000 P caf4f1ccbc684b5ca79d6a100cec1e37 [term 2 LEADER]: Becoming Leader. State: Replica: caf4f1ccbc684b5ca79d6a100cec1e37, State: Running, Role: LEADER
I20260621 05:12:52.993363 16396 sys_catalog.cc:564] T 00000000000000000000000000000000 P caf4f1ccbc684b5ca79d6a100cec1e37 [sys.catalog]: configured and running, proceeding with master startup.
I20260621 05:12:52.993495 16400 consensus_queue.cc:237] T 00000000000000000000000000000000 P caf4f1ccbc684b5ca79d6a100cec1e37 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "caf4f1ccbc684b5ca79d6a100cec1e37" member_type: VOTER last_known_addr { host: "127.3.131.190" port: 34691 } }
I20260621 05:12:52.993851 16400 sys_catalog.cc:455] T 00000000000000000000000000000000 P caf4f1ccbc684b5ca79d6a100cec1e37 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "caf4f1ccbc684b5ca79d6a100cec1e37" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "caf4f1ccbc684b5ca79d6a100cec1e37" member_type: VOTER last_known_addr { host: "127.3.131.190" port: 34691 } } }
I20260621 05:12:52.994017 16400 sys_catalog.cc:458] T 00000000000000000000000000000000 P caf4f1ccbc684b5ca79d6a100cec1e37 [sys.catalog]: This master's current role is: LEADER
I20260621 05:12:52.994189 16400 sys_catalog.cc:455] T 00000000000000000000000000000000 P caf4f1ccbc684b5ca79d6a100cec1e37 [sys.catalog]: SysCatalogTable state changed. Reason: New leader caf4f1ccbc684b5ca79d6a100cec1e37. Latest consensus state: current_term: 2 leader_uuid: "caf4f1ccbc684b5ca79d6a100cec1e37" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "caf4f1ccbc684b5ca79d6a100cec1e37" member_type: VOTER last_known_addr { host: "127.3.131.190" port: 34691 } } }
I20260621 05:12:52.994294 16400 sys_catalog.cc:458] T 00000000000000000000000000000000 P caf4f1ccbc684b5ca79d6a100cec1e37 [sys.catalog]: This master's current role is: LEADER
I20260621 05:12:52.994902 16396 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782018616648578-3598-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260621 05:12:52.995253 16409 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260621 05:12:52.997874 16409 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260621 05:12:52.998170 16409 catalog_manager.cc:1259] Loaded cluster ID: 2bb938f402e54c78aef2f408b7fe7ab5
I20260621 05:12:52.998279 16409 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260621 05:12:53.000574 16409 catalog_manager.cc:1504] Loading token signing keys...
I20260621 05:12:53.000994 16409 catalog_manager.cc:5879] T 00000000000000000000000000000000 P caf4f1ccbc684b5ca79d6a100cec1e37: Loaded TSK: 0
I20260621 05:12:53.001370 16409 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260621 05:12:53.108946 16399 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 05:12:53.109233 16399 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 05:12:53.109279 16399 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 05:12:53.111080 16399 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 05:12:53.111141 16399 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 05:12:53.111346 16399 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.131.129
I20260621 05:12:53.113540 16399 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.131.148:45931
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782018616648578-3598-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782018616648578-3598-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.3.131.148:50409/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782018616648578-3598-0/krb5kdc/kudu_127.3.131.129.keytab
--principal=kudu/127.3.131.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.131.129:43057
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782018616648578-3598-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.131.129
--webserver_port=35389
--tserver_master_addrs=127.3.131.190:34691
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16399
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.131.129
--log_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782018616648578-3598-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 21 Jun 2026 04:16:16 UTC on ccycloud.quasar-xycaha.root.comops.site
build id 4457881
I20260621 05:12:53.113797 16399 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 05:12:53.114059 16399 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 05:12:53.120985 16436 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:12:53.121011 16434 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:12:53.121368 16433 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:12:53.122598 16399 server_base.cc:1034] running on GCE node
I20260621 05:12:53.122816 16399 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 05:12:53.123095 16399 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 05:12:53.124271 16399 hybrid_clock.cc:648] HybridClock initialized: now 1782018773124234 us; error 52 us; skew 500 ppm
Jun 21 05:12:53 dist-test-slave-vrq1 krb5kdc[15364](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782018773, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.131.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 05:12:53.127552 16399 init.cc:377] Logged in from keytab as kudu/127.3.131.129@KRBTEST.COM (short username kudu)
I20260621 05:12:53.128033 16399 webserver.cc:460] Webserver started at http://127.3.131.129:35389/ using document root <none> and password file <none>
I20260621 05:12:53.128298 16399 fs_manager.cc:359] Metadata directory not provided
I20260621 05:12:53.128362 16399 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jun 21 05:12:53 dist-test-slave-vrq1 krb5kdc[15364](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782018773, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.131.129@KRBTEST.COM for HTTP/127.3.131.148@KRBTEST.COM
I20260621 05:12:53.148676 16399 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.001s sys 0.000s
I20260621 05:12:53.149777 16444 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 05:12:53.149988 16399 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260621 05:12:53.150074 16399 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782018616648578-3598-0/minicluster-data/ts-0/data,/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782018616648578-3598-0/minicluster-data/ts-0/wal
uuid: "cddc12c2e002400ea87d54d68d4ceec6"
format_stamp: "Formatted at 2026-06-21 05:12:52 on dist-test-slave-vrq1"
server_key: "73b15661824e25ba58aca5d5b03ba788"
server_key_iv: "7ed46e59df40f18a389be04a660a6722"
server_key_version: "kuduclusterkey@0"
I20260621 05:12:53.150547 16399 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782018616648578-3598-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782018616648578-3598-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782018616648578-3598-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 05:12:53.173749 16399 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 05:12:53.179237 16399 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 05:12:53.179430 16399 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 05:12:53.179679 16399 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 05:12:53.180071 16399 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 05:12:53.180205 16399 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 05:12:53.180259 16399 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 05:12:53.180279 16399 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 05:12:53.186762 16399 rpc_server.cc:307] RPC server started. Bound to: 127.3.131.129:43057
I20260621 05:12:53.187326 16399 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782018616648578-3598-0/minicluster-data/ts-0/data/info.pb
I20260621 05:12:53.190414 3598 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWZOx2_/build/release/bin/kudu as pid 16399
I20260621 05:12:53.190714 3598 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWZOx2_/build/release/bin/kudu
/tmp/dist-test-taskWZOx2_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782018616648578-3598-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782018616648578-3598-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782018616648578-3598-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782018616648578-3598-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.3.131.148:50409/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.131.130:45271
--local_ip_for_outbound_sockets=127.3.131.130
--tserver_master_addrs=127.3.131.190:34691
--webserver_port=34175
--webserver_interface=127.3.131.130
--keytab_file=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782018616648578-3598-0/krb5kdc/kudu_127.3.131.130.keytab
--principal=kudu/127.3.131.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.131.148:45931
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.3.131.148:50409/kms with env {KRB5CCNAME=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782018616648578-3598-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782018616648578-3598-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782018616648578-3598-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260621 05:12:53.200289 16557 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.131.129:43057 every 8 connection(s)
Jun 21 05:12:53 dist-test-slave-vrq1 krb5kdc[15364](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782018773, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.131.129@KRBTEST.COM for kudu/127.3.131.190@KRBTEST.COM
I20260621 05:12:53.222016 16562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:12:53.201166 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.131.129:40035 (local address 127.3.131.190:34691)
0621 05:12:53.201408 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0621 05:12:53.201418 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0621 05:12:53.204889 (+ 3471us) server_negotiation.cc:407] Connection header received
0621 05:12:53.205847 (+ 958us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:12:53.205858 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:12:53.205923 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:12:53.206034 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:12:53.207123 (+ 1089us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:12:53.208456 (+ 1333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:12:53.210011 (+ 1555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:12:53.210620 (+ 609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:12:53.210691 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:12:53.217440 (+ 6749us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:12:53.217446 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:12:53.217446 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:12:53.217463 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:12:53.219696 (+ 2233us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:12:53.220338 (+ 642us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:12:53.220342 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:12:53.220343 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:12:53.220470 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:12:53.220799 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:12:53.220801 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:12:53.220801 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:12:53.221121 (+ 320us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:12:53.221184 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0621 05:12:53.221643 (+ 459us) server_negotiation.cc:299] Negotiation successful
0621 05:12:53.221789 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":190,"thread_start_us":111,"threads_started":1}
I20260621 05:12:53.222645 16561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:12:53.201080 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.131.190:34691 (local address 127.3.131.129:40035)
0621 05:12:53.204640 (+ 3560us) negotiation.cc:107] Waiting for socket to connect
0621 05:12:53.204716 (+ 76us) client_negotiation.cc:174] Beginning negotiation
0621 05:12:53.205579 (+ 863us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:12:53.206303 (+ 724us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:12:53.206307 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:12:53.206602 (+ 295us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:12:53.206957 (+ 355us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:12:53.206966 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:12:53.208594 (+ 1628us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:12:53.208595 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:12:53.209904 (+ 1309us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:12:53.209907 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:12:53.210835 (+ 928us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:12:53.210836 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:12:53.210964 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:12:53.211683 (+ 719us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 05:12:53.211695 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:12:53.217251 (+ 5556us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 05:12:53.219843 (+ 2592us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:12:53.219857 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:12:53.219860 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:12:53.220205 (+ 345us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:12:53.220554 (+ 349us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:12:53.220555 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:12:53.220555 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:12:53.220715 (+ 160us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:12:53.221251 (+ 536us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 05:12:53.221259 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 05:12:53.221433 (+ 174us) client_negotiation.cc:769] Sending connection context
0621 05:12:53.222240 (+ 807us) client_negotiation.cc:240] Negotiation successful
0621 05:12:53.222427 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3495,"thread_start_us":98,"threads_started":1}
I20260621 05:12:53.223531 16558 heartbeater.cc:347] Connected to a master server at 127.3.131.190:34691
I20260621 05:12:53.223644 16558 heartbeater.cc:464] Registering TS with master...
I20260621 05:12:53.223858 16558 heartbeater.cc:511] Master 127.3.131.190:34691 requested a full tablet report, sending...
W20260621 05:12:53.224727 16558 heartbeater.cc:644] Failed to heartbeat to 127.3.131.190:34691 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260621 05:12:53.224815 16558 heartbeater.cc:464] Registering TS with master...
I20260621 05:12:53.224886 16558 heartbeater.cc:511] Master 127.3.131.190:34691 requested a full tablet report, sending...
I20260621 05:12:53.225203 16558 heartbeater.cc:464] Registering TS with master...
I20260621 05:12:53.225291 16558 heartbeater.cc:511] Master 127.3.131.190:34691 requested a full tablet report, sending...
W20260621 05:12:53.225572 16558 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260621 05:12:53.318250 16560 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 05:12:53.318563 16560 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 05:12:53.318600 16560 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 05:12:53.320410 16560 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 05:12:53.320473 16560 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 05:12:53.320526 16560 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.131.130
I20260621 05:12:53.322106 16560 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.131.148:45931
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782018616648578-3598-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782018616648578-3598-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.3.131.148:50409/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782018616648578-3598-0/krb5kdc/kudu_127.3.131.130.keytab
--principal=kudu/127.3.131.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.131.130:45271
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782018616648578-3598-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.131.130
--webserver_port=34175
--tserver_master_addrs=127.3.131.190:34691
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16560
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.131.130
--log_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782018616648578-3598-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 21 Jun 2026 04:16:16 UTC on ccycloud.quasar-xycaha.root.comops.site
build id 4457881
I20260621 05:12:53.322361 16560 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 05:12:53.322587 16560 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 05:12:53.328634 16567 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:12:53.328516 16570 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:12:53.328531 16568 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:12:53.329190 16560 server_base.cc:1034] running on GCE node
I20260621 05:12:53.329411 16560 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 05:12:53.329685 16560 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 05:12:53.330863 16560 hybrid_clock.cc:648] HybridClock initialized: now 1782018773330829 us; error 46 us; skew 500 ppm
Jun 21 05:12:53 dist-test-slave-vrq1 krb5kdc[15364](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782018773, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.131.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 05:12:53.334168 16560 init.cc:377] Logged in from keytab as kudu/127.3.131.130@KRBTEST.COM (short username kudu)
I20260621 05:12:53.334717 16560 webserver.cc:460] Webserver started at http://127.3.131.130:34175/ using document root <none> and password file <none>
I20260621 05:12:53.334995 16560 fs_manager.cc:359] Metadata directory not provided
I20260621 05:12:53.335070 16560 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jun 21 05:12:53 dist-test-slave-vrq1 krb5kdc[15364](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782018773, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.131.130@KRBTEST.COM for HTTP/127.3.131.148@KRBTEST.COM
I20260621 05:12:53.359481 16560 fs_manager.cc:709] Time spent opening directory manager: real 0.008s user 0.001s sys 0.000s
I20260621 05:12:53.360728 16578 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 05:12:53.360922 16560 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260621 05:12:53.361023 16560 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782018616648578-3598-0/minicluster-data/ts-1/data,/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782018616648578-3598-0/minicluster-data/ts-1/wal
uuid: "a3fbf94490ec4c3d8e35edf5aaaef40a"
format_stamp: "Formatted at 2026-06-21 05:12:52 on dist-test-slave-vrq1"
server_key: "1fd2eccf9b69f8673872a9ddad22a23f"
server_key_iv: "7b13a0fc7f016c7f9e602db7be5b4821"
server_key_version: "kuduclusterkey@0"
I20260621 05:12:53.361544 16560 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782018616648578-3598-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782018616648578-3598-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782018616648578-3598-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 05:12:53.373302 16560 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 05:12:53.375623 16560 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 05:12:53.375803 16560 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 05:12:53.376071 16560 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 05:12:53.376446 16560 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 05:12:53.376497 16560 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 05:12:53.376541 16560 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 05:12:53.376567 16560 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 05:12:53.383628 16560 rpc_server.cc:307] RPC server started. Bound to: 127.3.131.130:45271
I20260621 05:12:53.383711 16691 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.131.130:45271 every 8 connection(s)
I20260621 05:12:53.384030 16560 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782018616648578-3598-0/minicluster-data/ts-1/data/info.pb
I20260621 05:12:53.384578 3598 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWZOx2_/build/release/bin/kudu as pid 16560
I20260621 05:12:53.384793 3598 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWZOx2_/build/release/bin/kudu
/tmp/dist-test-taskWZOx2_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782018616648578-3598-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782018616648578-3598-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782018616648578-3598-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782018616648578-3598-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.3.131.148:50409/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.131.131:33035
--local_ip_for_outbound_sockets=127.3.131.131
--tserver_master_addrs=127.3.131.190:34691
--webserver_port=43025
--webserver_interface=127.3.131.131
--keytab_file=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782018616648578-3598-0/krb5kdc/kudu_127.3.131.131.keytab
--principal=kudu/127.3.131.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.131.148:45931
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.3.131.148:50409/kms with env {KRB5CCNAME=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782018616648578-3598-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782018616648578-3598-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782018616648578-3598-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 21 05:12:53 dist-test-slave-vrq1 krb5kdc[15364](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782018773, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.131.130@KRBTEST.COM for kudu/127.3.131.190@KRBTEST.COM
I20260621 05:12:53.395676 16562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:12:53.385274 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.131.130:35667 (local address 127.3.131.190:34691)
0621 05:12:53.385322 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0621 05:12:53.385324 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:12:53.385336 (+ 12us) server_negotiation.cc:407] Connection header received
0621 05:12:53.385855 (+ 519us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:12:53.385857 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:12:53.385874 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:12:53.385946 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:12:53.386638 (+ 692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:12:53.387139 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:12:53.388729 (+ 1590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:12:53.389056 (+ 327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:12:53.389137 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:12:53.391896 (+ 2759us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:12:53.391897 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:12:53.391898 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:12:53.391908 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:12:53.393769 (+ 1861us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:12:53.394376 (+ 607us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:12:53.394377 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:12:53.394378 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:12:53.394457 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:12:53.394769 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:12:53.394770 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:12:53.394770 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:12:53.394936 (+ 166us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:12:53.394983 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0621 05:12:53.395466 (+ 483us) server_negotiation.cc:299] Negotiation successful
0621 05:12:53.395572 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260621 05:12:53.396377 16694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:12:53.384766 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.131.190:34691 (local address 127.3.131.130:35667)
0621 05:12:53.384961 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0621 05:12:53.385009 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0621 05:12:53.385717 (+ 708us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:12:53.386015 (+ 298us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:12:53.386020 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:12:53.386276 (+ 256us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:12:53.386540 (+ 264us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:12:53.386545 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:12:53.387286 (+ 741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:12:53.387286 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:12:53.388628 (+ 1342us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:12:53.388632 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:12:53.389231 (+ 599us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:12:53.389232 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:12:53.389400 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:12:53.390056 (+ 656us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 05:12:53.390067 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:12:53.391771 (+ 1704us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 05:12:53.393934 (+ 2163us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:12:53.393942 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:12:53.393944 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:12:53.394283 (+ 339us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:12:53.394536 (+ 253us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:12:53.394536 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:12:53.394537 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:12:53.394679 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:12:53.395054 (+ 375us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 05:12:53.395061 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 05:12:53.395271 (+ 210us) client_negotiation.cc:769] Sending connection context
0621 05:12:53.395951 (+ 680us) client_negotiation.cc:240] Negotiation successful
0621 05:12:53.396135 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":91,"threads_started":1}
I20260621 05:12:53.397008 16692 heartbeater.cc:347] Connected to a master server at 127.3.131.190:34691
I20260621 05:12:53.397110 16692 heartbeater.cc:464] Registering TS with master...
I20260621 05:12:53.397293 16692 heartbeater.cc:511] Master 127.3.131.190:34691 requested a full tablet report, sending...
W20260621 05:12:53.397894 16692 heartbeater.cc:644] Failed to heartbeat to 127.3.131.190:34691 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260621 05:12:53.397980 16692 heartbeater.cc:464] Registering TS with master...
I20260621 05:12:53.398051 16692 heartbeater.cc:511] Master 127.3.131.190:34691 requested a full tablet report, sending...
I20260621 05:12:53.398358 16692 heartbeater.cc:464] Registering TS with master...
I20260621 05:12:53.398451 16692 heartbeater.cc:511] Master 127.3.131.190:34691 requested a full tablet report, sending...
W20260621 05:12:53.398741 16692 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260621 05:12:53.495522 16695 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 05:12:53.495819 16695 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 05:12:53.495865 16695 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 05:12:53.498027 16695 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 05:12:53.498118 16695 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 05:12:53.498167 16695 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.131.131
I20260621 05:12:53.501171 16695 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.131.148:45931
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782018616648578-3598-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782018616648578-3598-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.3.131.148:50409/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782018616648578-3598-0/krb5kdc/kudu_127.3.131.131.keytab
--principal=kudu/127.3.131.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.131.131:33035
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782018616648578-3598-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.131.131
--webserver_port=43025
--tserver_master_addrs=127.3.131.190:34691
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16695
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.131.131
--log_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782018616648578-3598-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 21 Jun 2026 04:16:16 UTC on ccycloud.quasar-xycaha.root.comops.site
build id 4457881
I20260621 05:12:53.501502 16695 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 05:12:53.501834 16695 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 05:12:53.508862 16703 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:12:53.508878 16700 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:12:53.508878 16701 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:12:53.509476 16695 server_base.cc:1034] running on GCE node
I20260621 05:12:53.509857 16695 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 05:12:53.510211 16695 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 05:12:53.511382 16695 hybrid_clock.cc:648] HybridClock initialized: now 1782018773511329 us; error 68 us; skew 500 ppm
Jun 21 05:12:53 dist-test-slave-vrq1 krb5kdc[15364](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782018773, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.131.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 05:12:53.514894 16695 init.cc:377] Logged in from keytab as kudu/127.3.131.131@KRBTEST.COM (short username kudu)
I20260621 05:12:53.515576 16695 webserver.cc:460] Webserver started at http://127.3.131.131:43025/ using document root <none> and password file <none>
I20260621 05:12:53.515949 16695 fs_manager.cc:359] Metadata directory not provided
I20260621 05:12:53.516093 16695 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jun 21 05:12:53 dist-test-slave-vrq1 krb5kdc[15364](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782018773, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.131.131@KRBTEST.COM for HTTP/127.3.131.148@KRBTEST.COM
I20260621 05:12:53.535077 16695 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.000s sys 0.002s
I20260621 05:12:53.536381 16711 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 05:12:53.536558 16695 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260621 05:12:53.536638 16695 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782018616648578-3598-0/minicluster-data/ts-2/data,/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782018616648578-3598-0/minicluster-data/ts-2/wal
uuid: "4644e1517d854a16a6716ea38148e104"
format_stamp: "Formatted at 2026-06-21 05:12:52 on dist-test-slave-vrq1"
server_key: "fc74f8844f70f259a73bc0386d6e1ef3"
server_key_iv: "3f44ac5c541a50708d45eef90f9934a9"
server_key_version: "kuduclusterkey@0"
I20260621 05:12:53.537150 16695 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782018616648578-3598-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782018616648578-3598-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782018616648578-3598-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 05:12:53.560070 16695 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 05:12:53.563784 16695 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 05:12:53.563983 16695 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 05:12:53.564275 16695 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 05:12:53.564694 16695 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 05:12:53.564749 16695 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 05:12:53.564867 16695 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 05:12:53.564898 16695 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 05:12:53.573276 16695 rpc_server.cc:307] RPC server started. Bound to: 127.3.131.131:33035
I20260621 05:12:53.573377 16825 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.131.131:33035 every 8 connection(s)
I20260621 05:12:53.573856 16695 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782018616648578-3598-0/minicluster-data/ts-2/data/info.pb
I20260621 05:12:53.582054 3598 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWZOx2_/build/release/bin/kudu as pid 16695
Jun 21 05:12:53 dist-test-slave-vrq1 krb5kdc[15364](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782018773, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.131.131@KRBTEST.COM for kudu/127.3.131.190@KRBTEST.COM
I20260621 05:12:53.587216 16562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:12:53.575145 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.131.131:57169 (local address 127.3.131.190:34691)
0621 05:12:53.575253 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0621 05:12:53.575255 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:12:53.575484 (+ 229us) server_negotiation.cc:407] Connection header received
0621 05:12:53.576164 (+ 680us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:12:53.576165 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:12:53.576180 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:12:53.576250 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:12:53.577290 (+ 1040us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:12:53.578076 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:12:53.579654 (+ 1578us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:12:53.579965 (+ 311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:12:53.580051 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:12:53.583253 (+ 3202us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:12:53.583255 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:12:53.583256 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:12:53.583267 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:12:53.585333 (+ 2066us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:12:53.585944 (+ 611us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:12:53.585945 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:12:53.585946 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:12:53.586018 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:12:53.586361 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:12:53.586362 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:12:53.586363 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:12:53.586504 (+ 141us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:12:53.586550 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0621 05:12:53.586956 (+ 406us) server_negotiation.cc:299] Negotiation successful
0621 05:12:53.587058 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260621 05:12:53.587690 16828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:12:53.575110 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.131.190:34691 (local address 127.3.131.131:57169)
0621 05:12:53.575351 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0621 05:12:53.575390 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0621 05:12:53.576037 (+ 647us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:12:53.576578 (+ 541us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:12:53.576584 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:12:53.576811 (+ 227us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:12:53.577134 (+ 323us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:12:53.577142 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:12:53.578207 (+ 1065us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:12:53.578211 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:12:53.579544 (+ 1333us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:12:53.579548 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:12:53.580152 (+ 604us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:12:53.580154 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:12:53.580310 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:12:53.580970 (+ 660us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 05:12:53.580982 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:12:53.583047 (+ 2065us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 05:12:53.585449 (+ 2402us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:12:53.585461 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:12:53.585464 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:12:53.585840 (+ 376us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:12:53.586094 (+ 254us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:12:53.586095 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:12:53.586095 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:12:53.586258 (+ 163us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:12:53.586598 (+ 340us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 05:12:53.586606 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 05:12:53.586796 (+ 190us) client_negotiation.cc:769] Sending connection context
0621 05:12:53.586920 (+ 124us) client_negotiation.cc:240] Negotiation successful
0621 05:12:53.587107 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":194,"thread_start_us":157,"threads_started":1}
I20260621 05:12:53.588550 16826 heartbeater.cc:347] Connected to a master server at 127.3.131.190:34691
I20260621 05:12:53.588665 16826 heartbeater.cc:464] Registering TS with master...
I20260621 05:12:53.588913 16826 heartbeater.cc:511] Master 127.3.131.190:34691 requested a full tablet report, sending...
W20260621 05:12:53.589877 16826 heartbeater.cc:644] Failed to heartbeat to 127.3.131.190:34691 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260621 05:12:53.589970 16826 heartbeater.cc:464] Registering TS with master...
I20260621 05:12:53.590054 16826 heartbeater.cc:511] Master 127.3.131.190:34691 requested a full tablet report, sending...
I20260621 05:12:53.590372 16826 heartbeater.cc:464] Registering TS with master...
I20260621 05:12:53.590451 16826 heartbeater.cc:511] Master 127.3.131.190:34691 requested a full tablet report, sending...
W20260621 05:12:53.590723 16826 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260621 05:12:53.592235 16829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:12:53.582534 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42192 (local address 127.3.131.190:34691)
0621 05:12:53.582886 (+ 352us) server_negotiation.cc:206] Beginning negotiation
0621 05:12:53.582887 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0621 05:12:53.582893 (+ 6us) server_negotiation.cc:407] Connection header received
0621 05:12:53.582917 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:12:53.582917 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:12:53.582928 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:12:53.583008 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:12:53.584974 (+ 1966us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:12:53.585753 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:12:53.586712 (+ 959us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:12:53.586998 (+ 286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:12:53.587064 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:12:53.588865 (+ 1801us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:12:53.588866 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:12:53.588867 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:12:53.588875 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:12:53.590902 (+ 2027us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:12:53.591297 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:12:53.591298 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:12:53.591298 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:12:53.591365 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:12:53.591576 (+ 211us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:12:53.591577 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:12:53.591577 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:12:53.591721 (+ 144us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:12:53.591888 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0621 05:12:53.592035 (+ 147us) server_negotiation.cc:299] Negotiation successful
0621 05:12:53.592131 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":330,"thread_start_us":92,"threads_started":1}
Jun 21 05:12:54 dist-test-slave-vrq1 krb5kdc[15364](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1782018774, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.131.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 05:12:54.225821 16558 heartbeater.cc:464] Registering TS with master...
I20260621 05:12:54.225980 16558 heartbeater.cc:511] Master 127.3.131.190:34691 requested a full tablet report, sending...
I20260621 05:12:54.226531 16360 ts_manager.cc:194] Registered new tserver with Master: cddc12c2e002400ea87d54d68d4ceec6 (127.3.131.129:43057)
I20260621 05:12:54.227584 16360 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.131.129@KRBTEST.COM'} at 127.3.131.129:40035
I20260621 05:12:54.398936 16692 heartbeater.cc:464] Registering TS with master...
I20260621 05:12:54.399106 16692 heartbeater.cc:511] Master 127.3.131.190:34691 requested a full tablet report, sending...
I20260621 05:12:54.399448 16360 ts_manager.cc:194] Registered new tserver with Master: a3fbf94490ec4c3d8e35edf5aaaef40a (127.3.131.130:45271)
I20260621 05:12:54.399977 16360 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.131.130@KRBTEST.COM'} at 127.3.131.130:35667
Jun 21 05:12:54 dist-test-slave-vrq1 krb5kdc[15364](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782018774, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.131.190@KRBTEST.COM for HTTP/127.3.131.148@KRBTEST.COM
I20260621 05:12:54.590931 16826 heartbeater.cc:464] Registering TS with master...
I20260621 05:12:54.591091 16826 heartbeater.cc:511] Master 127.3.131.190:34691 requested a full tablet report, sending...
I20260621 05:12:54.591475 16360 ts_manager.cc:194] Registered new tserver with Master: 4644e1517d854a16a6716ea38148e104 (127.3.131.131:33035)
I20260621 05:12:54.592017 16360 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.131.131@KRBTEST.COM'} at 127.3.131.131:57169
I20260621 05:12:54.596133 3598 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260621 05:12:54.606351 16848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:12:54.597272 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42204 (local address 127.3.131.190:34691)
0621 05:12:54.598449 (+ 1177us) server_negotiation.cc:206] Beginning negotiation
0621 05:12:54.598452 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0621 05:12:54.598462 (+ 10us) server_negotiation.cc:407] Connection header received
0621 05:12:54.598492 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:12:54.598492 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:12:54.598505 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:12:54.598580 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:12:54.599524 (+ 944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:12:54.600152 (+ 628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:12:54.600989 (+ 837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:12:54.601293 (+ 304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:12:54.601378 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:12:54.602875 (+ 1497us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:12:54.602877 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:12:54.602877 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:12:54.602886 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:12:54.604746 (+ 1860us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:12:54.605173 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:12:54.605175 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:12:54.605175 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:12:54.605241 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:12:54.605435 (+ 194us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:12:54.605436 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:12:54.605436 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:12:54.605577 (+ 141us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:12:54.605668 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0621 05:12:54.606148 (+ 480us) server_negotiation.cc:299] Negotiation successful
0621 05:12:54.606233 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1148,"thread_start_us":68,"threads_started":1}
I20260621 05:12:54.608966 16360 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42204:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260621 05:12:54.611166 16360 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260621 05:12:54.624460 16854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:12:54.617708 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.131.131:33035 (local address 127.0.0.1:35568)
0621 05:12:54.618092 (+ 384us) negotiation.cc:107] Waiting for socket to connect
0621 05:12:54.618102 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0621 05:12:54.618139 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:12:54.618456 (+ 317us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:12:54.618460 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:12:54.618462 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 05:12:54.618547 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:12:54.618549 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:12:54.619984 (+ 1435us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:12:54.619985 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:12:54.622754 (+ 2769us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:12:54.622758 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:12:54.624155 (+ 1397us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:12:54.624156 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:12:54.624246 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:12:54.624267 (+ 21us) client_negotiation.cc:769] Sending connection context
0621 05:12:54.624319 (+ 52us) client_negotiation.cc:240] Negotiation successful
0621 05:12:54.624380 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":367,"thread_start_us":104,"threads_started":1}
I20260621 05:12:54.626792 16857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:12:54.618150 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35568 (local address 127.3.131.131:33035)
0621 05:12:54.618269 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0621 05:12:54.618274 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 05:12:54.618285 (+ 11us) server_negotiation.cc:407] Connection header received
0621 05:12:54.618321 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:12:54.618323 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:12:54.618339 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:12:54.618448 (+ 109us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 05:12:54.618632 (+ 184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:12:54.619616 (+ 984us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:12:54.622869 (+ 3253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:12:54.624042 (+ 1173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:12:54.625172 (+ 1130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:12:54.626381 (+ 1209us) server_negotiation.cc:1035] Waiting for connection context
0621 05:12:54.626650 (+ 269us) server_negotiation.cc:299] Negotiation successful
0621 05:12:54.626708 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82,"thread_start_us":49,"threads_started":1}
I20260621 05:12:54.628051 16760 tablet_service.cc:1467] Processing CreateTablet for tablet 2adae2fa6e294bf78ee0a8ba20cf8911 (DEFAULT_TABLE table=test-table [id=b1c7d5b7fab349acb49b62be3993b94f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 05:12:54.628450 16760 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2adae2fa6e294bf78ee0a8ba20cf8911. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 05:12:54.628736 16855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:12:54.617970 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.131.129:43057 (local address 127.0.0.1:45110)
0621 05:12:54.618662 (+ 692us) negotiation.cc:107] Waiting for socket to connect
0621 05:12:54.618673 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0621 05:12:54.618718 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:12:54.623560 (+ 4842us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:12:54.623560 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:12:54.623563 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 05:12:54.623597 (+ 34us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:12:54.623598 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:12:54.625057 (+ 1459us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:12:54.625058 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:12:54.626217 (+ 1159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:12:54.626220 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:12:54.628471 (+ 2251us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:12:54.628472 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:12:54.628537 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:12:54.628543 (+ 6us) client_negotiation.cc:769] Sending connection context
0621 05:12:54.628610 (+ 67us) client_negotiation.cc:240] Negotiation successful
0621 05:12:54.628655 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":671,"mutex_wait_us":22,"thread_start_us":72,"threads_started":1}
I20260621 05:12:54.628750 16858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:12:54.618180 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45110 (local address 127.3.131.129:43057)
0621 05:12:54.623305 (+ 5125us) server_negotiation.cc:206] Beginning negotiation
0621 05:12:54.623311 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 05:12:54.623324 (+ 13us) server_negotiation.cc:407] Connection header received
0621 05:12:54.623363 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:12:54.623365 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:12:54.623383 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:12:54.623470 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 05:12:54.623673 (+ 203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:12:54.624969 (+ 1296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:12:54.627589 (+ 2620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:12:54.628370 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:12:54.628444 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:12:54.628500 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0621 05:12:54.628630 (+ 130us) server_negotiation.cc:299] Negotiation successful
0621 05:12:54.628668 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5083,"thread_start_us":75,"threads_started":1}
I20260621 05:12:54.631608 16859 tablet_bootstrap.cc:492] T 2adae2fa6e294bf78ee0a8ba20cf8911 P 4644e1517d854a16a6716ea38148e104: Bootstrap starting.
I20260621 05:12:54.632334 16859 tablet_bootstrap.cc:654] T 2adae2fa6e294bf78ee0a8ba20cf8911 P 4644e1517d854a16a6716ea38148e104: Neither blocks nor log segments found. Creating new log.
I20260621 05:12:54.632489 16853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:12:54.617702 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.131.130:45271 (local address 127.0.0.1:36276)
0621 05:12:54.617871 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0621 05:12:54.617914 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0621 05:12:54.617978 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:12:54.623502 (+ 5524us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:12:54.623504 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:12:54.623506 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 05:12:54.623554 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:12:54.623556 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:12:54.624888 (+ 1332us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:12:54.624889 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:12:54.627470 (+ 2581us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:12:54.627472 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:12:54.632225 (+ 4753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:12:54.632226 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:12:54.632307 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:12:54.632313 (+ 6us) client_negotiation.cc:769] Sending connection context
0621 05:12:54.632345 (+ 32us) client_negotiation.cc:240] Negotiation successful
0621 05:12:54.632399 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":134,"mutex_wait_us":1099,"thread_start_us":102,"threads_started":1}
I20260621 05:12:54.632640 16859 log.cc:824] T 2adae2fa6e294bf78ee0a8ba20cf8911 P 4644e1517d854a16a6716ea38148e104: Log is configured to *not* fsync() on all Append() calls
I20260621 05:12:54.632588 16492 tablet_service.cc:1467] Processing CreateTablet for tablet 2adae2fa6e294bf78ee0a8ba20cf8911 (DEFAULT_TABLE table=test-table [id=b1c7d5b7fab349acb49b62be3993b94f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 05:12:54.632926 16856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:12:54.618002 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36276 (local address 127.3.131.130:45271)
0621 05:12:54.623305 (+ 5303us) server_negotiation.cc:206] Beginning negotiation
0621 05:12:54.623311 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 05:12:54.623324 (+ 13us) server_negotiation.cc:407] Connection header received
0621 05:12:54.623363 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:12:54.623365 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:12:54.623383 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:12:54.623630 (+ 247us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 05:12:54.623717 (+ 87us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:12:54.624768 (+ 1051us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:12:54.631270 (+ 6502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:12:54.632110 (+ 840us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:12:54.632671 (+ 561us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:12:54.632762 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0621 05:12:54.632802 (+ 40us) server_negotiation.cc:299] Negotiation successful
0621 05:12:54.632851 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5252,"thread_start_us":66,"threads_started":1}
I20260621 05:12:54.632947 16492 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2adae2fa6e294bf78ee0a8ba20cf8911. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 05:12:54.633611 16859 tablet_bootstrap.cc:492] T 2adae2fa6e294bf78ee0a8ba20cf8911 P 4644e1517d854a16a6716ea38148e104: No bootstrap required, opened a new log
I20260621 05:12:54.633675 16859 ts_tablet_manager.cc:1397] T 2adae2fa6e294bf78ee0a8ba20cf8911 P 4644e1517d854a16a6716ea38148e104: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260621 05:12:54.633586 16613 tablet_service.cc:1467] Processing CreateTablet for tablet 2adae2fa6e294bf78ee0a8ba20cf8911 (DEFAULT_TABLE table=test-table [id=b1c7d5b7fab349acb49b62be3993b94f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 05:12:54.633911 16613 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2adae2fa6e294bf78ee0a8ba20cf8911. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 05:12:54.636886 16859 raft_consensus.cc:348] T 2adae2fa6e294bf78ee0a8ba20cf8911 P 4644e1517d854a16a6716ea38148e104 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3fbf94490ec4c3d8e35edf5aaaef40a" member_type: VOTER last_known_addr { host: "127.3.131.130" port: 45271 } } peers { permanent_uuid: "4644e1517d854a16a6716ea38148e104" member_type: VOTER last_known_addr { host: "127.3.131.131" port: 33035 } } peers { permanent_uuid: "cddc12c2e002400ea87d54d68d4ceec6" member_type: VOTER last_known_addr { host: "127.3.131.129" port: 43057 } }
I20260621 05:12:54.637061 16859 raft_consensus.cc:374] T 2adae2fa6e294bf78ee0a8ba20cf8911 P 4644e1517d854a16a6716ea38148e104 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 05:12:54.637091 16859 raft_consensus.cc:729] T 2adae2fa6e294bf78ee0a8ba20cf8911 P 4644e1517d854a16a6716ea38148e104 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4644e1517d854a16a6716ea38148e104, State: Initialized, Role: FOLLOWER
I20260621 05:12:54.637234 16859 consensus_queue.cc:260] T 2adae2fa6e294bf78ee0a8ba20cf8911 P 4644e1517d854a16a6716ea38148e104 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3fbf94490ec4c3d8e35edf5aaaef40a" member_type: VOTER last_known_addr { host: "127.3.131.130" port: 45271 } } peers { permanent_uuid: "4644e1517d854a16a6716ea38148e104" member_type: VOTER last_known_addr { host: "127.3.131.131" port: 33035 } } peers { permanent_uuid: "cddc12c2e002400ea87d54d68d4ceec6" member_type: VOTER last_known_addr { host: "127.3.131.129" port: 43057 } }
I20260621 05:12:54.637576 16859 ts_tablet_manager.cc:1428] T 2adae2fa6e294bf78ee0a8ba20cf8911 P 4644e1517d854a16a6716ea38148e104: Time spent starting tablet: real 0.004s user 0.000s sys 0.004s
I20260621 05:12:54.637815 16826 heartbeater.cc:503] Master 127.3.131.190:34691 was elected leader, sending a full tablet report...
I20260621 05:12:54.639380 16861 tablet_bootstrap.cc:492] T 2adae2fa6e294bf78ee0a8ba20cf8911 P cddc12c2e002400ea87d54d68d4ceec6: Bootstrap starting.
I20260621 05:12:54.640143 16861 tablet_bootstrap.cc:654] T 2adae2fa6e294bf78ee0a8ba20cf8911 P cddc12c2e002400ea87d54d68d4ceec6: Neither blocks nor log segments found. Creating new log.
I20260621 05:12:54.640494 16861 log.cc:824] T 2adae2fa6e294bf78ee0a8ba20cf8911 P cddc12c2e002400ea87d54d68d4ceec6: Log is configured to *not* fsync() on all Append() calls
I20260621 05:12:54.640961 16862 tablet_bootstrap.cc:492] T 2adae2fa6e294bf78ee0a8ba20cf8911 P a3fbf94490ec4c3d8e35edf5aaaef40a: Bootstrap starting.
I20260621 05:12:54.641774 16862 tablet_bootstrap.cc:654] T 2adae2fa6e294bf78ee0a8ba20cf8911 P a3fbf94490ec4c3d8e35edf5aaaef40a: Neither blocks nor log segments found. Creating new log.
I20260621 05:12:54.642169 16862 log.cc:824] T 2adae2fa6e294bf78ee0a8ba20cf8911 P a3fbf94490ec4c3d8e35edf5aaaef40a: Log is configured to *not* fsync() on all Append() calls
I20260621 05:12:54.644094 16861 tablet_bootstrap.cc:492] T 2adae2fa6e294bf78ee0a8ba20cf8911 P cddc12c2e002400ea87d54d68d4ceec6: No bootstrap required, opened a new log
I20260621 05:12:54.644181 16861 ts_tablet_manager.cc:1397] T 2adae2fa6e294bf78ee0a8ba20cf8911 P cddc12c2e002400ea87d54d68d4ceec6: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20260621 05:12:54.647332 16861 raft_consensus.cc:348] T 2adae2fa6e294bf78ee0a8ba20cf8911 P cddc12c2e002400ea87d54d68d4ceec6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3fbf94490ec4c3d8e35edf5aaaef40a" member_type: VOTER last_known_addr { host: "127.3.131.130" port: 45271 } } peers { permanent_uuid: "4644e1517d854a16a6716ea38148e104" member_type: VOTER last_known_addr { host: "127.3.131.131" port: 33035 } } peers { permanent_uuid: "cddc12c2e002400ea87d54d68d4ceec6" member_type: VOTER last_known_addr { host: "127.3.131.129" port: 43057 } }
I20260621 05:12:54.647604 16861 raft_consensus.cc:374] T 2adae2fa6e294bf78ee0a8ba20cf8911 P cddc12c2e002400ea87d54d68d4ceec6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 05:12:54.647648 16861 raft_consensus.cc:729] T 2adae2fa6e294bf78ee0a8ba20cf8911 P cddc12c2e002400ea87d54d68d4ceec6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cddc12c2e002400ea87d54d68d4ceec6, State: Initialized, Role: FOLLOWER
I20260621 05:12:54.647864 16861 consensus_queue.cc:260] T 2adae2fa6e294bf78ee0a8ba20cf8911 P cddc12c2e002400ea87d54d68d4ceec6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3fbf94490ec4c3d8e35edf5aaaef40a" member_type: VOTER last_known_addr { host: "127.3.131.130" port: 45271 } } peers { permanent_uuid: "4644e1517d854a16a6716ea38148e104" member_type: VOTER last_known_addr { host: "127.3.131.131" port: 33035 } } peers { permanent_uuid: "cddc12c2e002400ea87d54d68d4ceec6" member_type: VOTER last_known_addr { host: "127.3.131.129" port: 43057 } }
I20260621 05:12:54.648311 16861 ts_tablet_manager.cc:1428] T 2adae2fa6e294bf78ee0a8ba20cf8911 P cddc12c2e002400ea87d54d68d4ceec6: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20260621 05:12:54.648854 16558 heartbeater.cc:503] Master 127.3.131.190:34691 was elected leader, sending a full tablet report...
I20260621 05:12:54.649806 16862 tablet_bootstrap.cc:492] T 2adae2fa6e294bf78ee0a8ba20cf8911 P a3fbf94490ec4c3d8e35edf5aaaef40a: No bootstrap required, opened a new log
I20260621 05:12:54.649899 16862 ts_tablet_manager.cc:1397] T 2adae2fa6e294bf78ee0a8ba20cf8911 P a3fbf94490ec4c3d8e35edf5aaaef40a: Time spent bootstrapping tablet: real 0.009s user 0.001s sys 0.000s
I20260621 05:12:54.652756 16862 raft_consensus.cc:348] T 2adae2fa6e294bf78ee0a8ba20cf8911 P a3fbf94490ec4c3d8e35edf5aaaef40a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3fbf94490ec4c3d8e35edf5aaaef40a" member_type: VOTER last_known_addr { host: "127.3.131.130" port: 45271 } } peers { permanent_uuid: "4644e1517d854a16a6716ea38148e104" member_type: VOTER last_known_addr { host: "127.3.131.131" port: 33035 } } peers { permanent_uuid: "cddc12c2e002400ea87d54d68d4ceec6" member_type: VOTER last_known_addr { host: "127.3.131.129" port: 43057 } }
I20260621 05:12:54.652979 16862 raft_consensus.cc:374] T 2adae2fa6e294bf78ee0a8ba20cf8911 P a3fbf94490ec4c3d8e35edf5aaaef40a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 05:12:54.653018 16862 raft_consensus.cc:729] T 2adae2fa6e294bf78ee0a8ba20cf8911 P a3fbf94490ec4c3d8e35edf5aaaef40a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a3fbf94490ec4c3d8e35edf5aaaef40a, State: Initialized, Role: FOLLOWER
I20260621 05:12:54.653191 16862 consensus_queue.cc:260] T 2adae2fa6e294bf78ee0a8ba20cf8911 P a3fbf94490ec4c3d8e35edf5aaaef40a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3fbf94490ec4c3d8e35edf5aaaef40a" member_type: VOTER last_known_addr { host: "127.3.131.130" port: 45271 } } peers { permanent_uuid: "4644e1517d854a16a6716ea38148e104" member_type: VOTER last_known_addr { host: "127.3.131.131" port: 33035 } } peers { permanent_uuid: "cddc12c2e002400ea87d54d68d4ceec6" member_type: VOTER last_known_addr { host: "127.3.131.129" port: 43057 } }
I20260621 05:12:54.653558 16692 heartbeater.cc:503] Master 127.3.131.190:34691 was elected leader, sending a full tablet report...
I20260621 05:12:54.653823 16862 ts_tablet_manager.cc:1428] T 2adae2fa6e294bf78ee0a8ba20cf8911 P a3fbf94490ec4c3d8e35edf5aaaef40a: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
W20260621 05:12:54.701820 16559 tablet.cc:2369] T 2adae2fa6e294bf78ee0a8ba20cf8911 P cddc12c2e002400ea87d54d68d4ceec6: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260621 05:12:54.747470 16866 raft_consensus.cc:482] T 2adae2fa6e294bf78ee0a8ba20cf8911 P cddc12c2e002400ea87d54d68d4ceec6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260621 05:12:54.747632 16866 raft_consensus.cc:504] T 2adae2fa6e294bf78ee0a8ba20cf8911 P cddc12c2e002400ea87d54d68d4ceec6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3fbf94490ec4c3d8e35edf5aaaef40a" member_type: VOTER last_known_addr { host: "127.3.131.130" port: 45271 } } peers { permanent_uuid: "4644e1517d854a16a6716ea38148e104" member_type: VOTER last_known_addr { host: "127.3.131.131" port: 33035 } } peers { permanent_uuid: "cddc12c2e002400ea87d54d68d4ceec6" member_type: VOTER last_known_addr { host: "127.3.131.129" port: 43057 } }
I20260621 05:12:54.748391 16866 leader_election.cc:288] T 2adae2fa6e294bf78ee0a8ba20cf8911 P cddc12c2e002400ea87d54d68d4ceec6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a3fbf94490ec4c3d8e35edf5aaaef40a (127.3.131.130:45271), 4644e1517d854a16a6716ea38148e104 (127.3.131.131:33035)
I20260621 05:12:54.752331 16857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:12:54.748511 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.131.129:47993 (local address 127.3.131.131:33035)
0621 05:12:54.748576 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0621 05:12:54.748578 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:12:54.748658 (+ 80us) server_negotiation.cc:407] Connection header received
0621 05:12:54.748788 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:12:54.748789 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:12:54.748800 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:12:54.748831 (+ 31us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 05:12:54.749080 (+ 249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:12:54.749812 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:12:54.751334 (+ 1522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:12:54.751915 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:12:54.751967 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:12:54.751976 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0621 05:12:54.752206 (+ 230us) server_negotiation.cc:299] Negotiation successful
0621 05:12:54.752251 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260621 05:12:54.752467 16856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:12:54.748644 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.131.129:51121 (local address 127.3.131.130:45271)
0621 05:12:54.748684 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0621 05:12:54.748686 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:12:54.748804 (+ 118us) server_negotiation.cc:407] Connection header received
0621 05:12:54.748911 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:12:54.748912 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:12:54.748920 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:12:54.748950 (+ 30us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 05:12:54.749154 (+ 204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:12:54.749911 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:12:54.751437 (+ 1526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:12:54.752036 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:12:54.752086 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:12:54.752095 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0621 05:12:54.752329 (+ 234us) server_negotiation.cc:299] Negotiation successful
0621 05:12:54.752373 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260621 05:12:54.752775 16868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:12:54.748238 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.131.131:33035 (local address 127.3.131.129:47993)
0621 05:12:54.748579 (+ 341us) negotiation.cc:107] Waiting for socket to connect
0621 05:12:54.748592 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0621 05:12:54.748690 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:12:54.748906 (+ 216us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:12:54.748908 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:12:54.748910 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 05:12:54.748993 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:12:54.748995 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:12:54.749920 (+ 925us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:12:54.749921 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:12:54.751224 (+ 1303us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:12:54.751228 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:12:54.752053 (+ 825us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:12:54.752054 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:12:54.752123 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:12:54.752127 (+ 4us) client_negotiation.cc:769] Sending connection context
0621 05:12:54.752611 (+ 484us) client_negotiation.cc:240] Negotiation successful
0621 05:12:54.752661 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":321,"thread_start_us":111,"threads_started":1}
I20260621 05:12:54.753199 16869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:12:54.748239 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.131.130:45271 (local address 127.3.131.129:51121)
0621 05:12:54.748737 (+ 498us) negotiation.cc:107] Waiting for socket to connect
0621 05:12:54.748748 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0621 05:12:54.748830 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:12:54.749015 (+ 185us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:12:54.749016 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:12:54.749017 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 05:12:54.749085 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:12:54.749087 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:12:54.750090 (+ 1003us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:12:54.750091 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:12:54.751329 (+ 1238us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:12:54.751333 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:12:54.752157 (+ 824us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:12:54.752158 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:12:54.752236 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:12:54.752239 (+ 3us) client_negotiation.cc:769] Sending connection context
0621 05:12:54.752798 (+ 559us) client_negotiation.cc:240] Negotiation successful
0621 05:12:54.753116 (+ 318us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":477,"thread_start_us":112,"threads_started":1}
I20260621 05:12:54.753257 16780 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2adae2fa6e294bf78ee0a8ba20cf8911" candidate_uuid: "cddc12c2e002400ea87d54d68d4ceec6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4644e1517d854a16a6716ea38148e104" is_pre_election: true
I20260621 05:12:54.753444 16780 raft_consensus.cc:2454] T 2adae2fa6e294bf78ee0a8ba20cf8911 P 4644e1517d854a16a6716ea38148e104 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cddc12c2e002400ea87d54d68d4ceec6 in term 0.
I20260621 05:12:54.753742 16447 leader_election.cc:302] T 2adae2fa6e294bf78ee0a8ba20cf8911 P cddc12c2e002400ea87d54d68d4ceec6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4644e1517d854a16a6716ea38148e104, cddc12c2e002400ea87d54d68d4ceec6; no voters:
I20260621 05:12:54.753736 16646 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2adae2fa6e294bf78ee0a8ba20cf8911" candidate_uuid: "cddc12c2e002400ea87d54d68d4ceec6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a3fbf94490ec4c3d8e35edf5aaaef40a" is_pre_election: true
I20260621 05:12:54.753888 16646 raft_consensus.cc:2454] T 2adae2fa6e294bf78ee0a8ba20cf8911 P a3fbf94490ec4c3d8e35edf5aaaef40a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cddc12c2e002400ea87d54d68d4ceec6 in term 0.
I20260621 05:12:54.753885 16866 raft_consensus.cc:2781] T 2adae2fa6e294bf78ee0a8ba20cf8911 P cddc12c2e002400ea87d54d68d4ceec6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260621 05:12:54.753922 16866 raft_consensus.cc:482] T 2adae2fa6e294bf78ee0a8ba20cf8911 P cddc12c2e002400ea87d54d68d4ceec6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260621 05:12:54.753945 16866 raft_consensus.cc:3037] T 2adae2fa6e294bf78ee0a8ba20cf8911 P cddc12c2e002400ea87d54d68d4ceec6 [term 0 FOLLOWER]: Advancing to term 1
I20260621 05:12:54.754961 16866 raft_consensus.cc:504] T 2adae2fa6e294bf78ee0a8ba20cf8911 P cddc12c2e002400ea87d54d68d4ceec6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3fbf94490ec4c3d8e35edf5aaaef40a" member_type: VOTER last_known_addr { host: "127.3.131.130" port: 45271 } } peers { permanent_uuid: "4644e1517d854a16a6716ea38148e104" member_type: VOTER last_known_addr { host: "127.3.131.131" port: 33035 } } peers { permanent_uuid: "cddc12c2e002400ea87d54d68d4ceec6" member_type: VOTER last_known_addr { host: "127.3.131.129" port: 43057 } }
I20260621 05:12:54.755101 16866 leader_election.cc:288] T 2adae2fa6e294bf78ee0a8ba20cf8911 P cddc12c2e002400ea87d54d68d4ceec6 [CANDIDATE]: Term 1 election: Requested vote from peers a3fbf94490ec4c3d8e35edf5aaaef40a (127.3.131.130:45271), 4644e1517d854a16a6716ea38148e104 (127.3.131.131:33035)
I20260621 05:12:54.755388 16646 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2adae2fa6e294bf78ee0a8ba20cf8911" candidate_uuid: "cddc12c2e002400ea87d54d68d4ceec6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a3fbf94490ec4c3d8e35edf5aaaef40a"
I20260621 05:12:54.755417 16780 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2adae2fa6e294bf78ee0a8ba20cf8911" candidate_uuid: "cddc12c2e002400ea87d54d68d4ceec6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4644e1517d854a16a6716ea38148e104"
I20260621 05:12:54.755481 16646 raft_consensus.cc:3037] T 2adae2fa6e294bf78ee0a8ba20cf8911 P a3fbf94490ec4c3d8e35edf5aaaef40a [term 0 FOLLOWER]: Advancing to term 1
I20260621 05:12:54.755538 16780 raft_consensus.cc:3037] T 2adae2fa6e294bf78ee0a8ba20cf8911 P 4644e1517d854a16a6716ea38148e104 [term 0 FOLLOWER]: Advancing to term 1
I20260621 05:12:54.756338 16646 raft_consensus.cc:2454] T 2adae2fa6e294bf78ee0a8ba20cf8911 P a3fbf94490ec4c3d8e35edf5aaaef40a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cddc12c2e002400ea87d54d68d4ceec6 in term 1.
I20260621 05:12:54.756531 16780 raft_consensus.cc:2454] T 2adae2fa6e294bf78ee0a8ba20cf8911 P 4644e1517d854a16a6716ea38148e104 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cddc12c2e002400ea87d54d68d4ceec6 in term 1.
I20260621 05:12:54.756727 16445 leader_election.cc:302] T 2adae2fa6e294bf78ee0a8ba20cf8911 P cddc12c2e002400ea87d54d68d4ceec6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a3fbf94490ec4c3d8e35edf5aaaef40a, cddc12c2e002400ea87d54d68d4ceec6; no voters:
I20260621 05:12:54.756958 16866 raft_consensus.cc:2781] T 2adae2fa6e294bf78ee0a8ba20cf8911 P cddc12c2e002400ea87d54d68d4ceec6 [term 1 FOLLOWER]: Leader election won for term 1
I20260621 05:12:54.757121 16866 raft_consensus.cc:686] T 2adae2fa6e294bf78ee0a8ba20cf8911 P cddc12c2e002400ea87d54d68d4ceec6 [term 1 LEADER]: Becoming Leader. State: Replica: cddc12c2e002400ea87d54d68d4ceec6, State: Running, Role: LEADER
I20260621 05:12:54.757216 16866 consensus_queue.cc:237] T 2adae2fa6e294bf78ee0a8ba20cf8911 P cddc12c2e002400ea87d54d68d4ceec6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3fbf94490ec4c3d8e35edf5aaaef40a" member_type: VOTER last_known_addr { host: "127.3.131.130" port: 45271 } } peers { permanent_uuid: "4644e1517d854a16a6716ea38148e104" member_type: VOTER last_known_addr { host: "127.3.131.131" port: 33035 } } peers { permanent_uuid: "cddc12c2e002400ea87d54d68d4ceec6" member_type: VOTER last_known_addr { host: "127.3.131.129" port: 43057 } }
I20260621 05:12:54.758045 16360 catalog_manager.cc:5495] T 2adae2fa6e294bf78ee0a8ba20cf8911 P cddc12c2e002400ea87d54d68d4ceec6 reported cstate change: term changed from 0 to 1, leader changed from <none> to cddc12c2e002400ea87d54d68d4ceec6 (127.3.131.129). New cstate: current_term: 1 leader_uuid: "cddc12c2e002400ea87d54d68d4ceec6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3fbf94490ec4c3d8e35edf5aaaef40a" member_type: VOTER last_known_addr { host: "127.3.131.130" port: 45271 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4644e1517d854a16a6716ea38148e104" member_type: VOTER last_known_addr { host: "127.3.131.131" port: 33035 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cddc12c2e002400ea87d54d68d4ceec6" member_type: VOTER last_known_addr { host: "127.3.131.129" port: 43057 } health_report { overall_health: HEALTHY } } }
I20260621 05:12:54.770133 16858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:12:54.766535 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45114 (local address 127.3.131.129:43057)
0621 05:12:54.766582 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0621 05:12:54.766584 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:12:54.766595 (+ 11us) server_negotiation.cc:407] Connection header received
0621 05:12:54.766665 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:12:54.766666 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:12:54.766689 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:12:54.766768 (+ 79us) server_negotiation.cc:226] Negotiated authn=TOKEN
0621 05:12:54.766977 (+ 209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:12:54.767797 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:12:54.768999 (+ 1202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:12:54.769334 (+ 335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:12:54.769398 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:12:54.769616 (+ 218us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0621 05:12:54.769766 (+ 150us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0621 05:12:54.769842 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0621 05:12:54.770026 (+ 184us) server_negotiation.cc:299] Negotiation successful
0621 05:12:54.770048 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260621 05:12:54.772071 16780 raft_consensus.cc:1261] T 2adae2fa6e294bf78ee0a8ba20cf8911 P 4644e1517d854a16a6716ea38148e104 [term 1 FOLLOWER]: Refusing update from remote peer cddc12c2e002400ea87d54d68d4ceec6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 05:12:54.772104 16646 raft_consensus.cc:1261] T 2adae2fa6e294bf78ee0a8ba20cf8911 P a3fbf94490ec4c3d8e35edf5aaaef40a [term 1 FOLLOWER]: Refusing update from remote peer cddc12c2e002400ea87d54d68d4ceec6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 05:12:54.772450 16866 consensus_queue.cc:1035] T 2adae2fa6e294bf78ee0a8ba20cf8911 P cddc12c2e002400ea87d54d68d4ceec6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a3fbf94490ec4c3d8e35edf5aaaef40a" member_type: VOTER last_known_addr { host: "127.3.131.130" port: 45271 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 05:12:54.772636 16870 consensus_queue.cc:1035] T 2adae2fa6e294bf78ee0a8ba20cf8911 P cddc12c2e002400ea87d54d68d4ceec6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4644e1517d854a16a6716ea38148e104" member_type: VOTER last_known_addr { host: "127.3.131.131" port: 33035 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 05:12:54.777863 16876 mvcc.cc:205] Tried to move back new op lower bound from 7299148901463724032 to 7299148901406150656. Current Snapshot: MvccSnapshot[applied={T|T < 7299148901463724032}]
I20260621 05:12:54.780208 16359 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42204:
table { table_name: "test-table" } modify_external_catalogs: true
I20260621 05:12:54.780325 16359 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42204:
table { table_name: "test-table" } modify_external_catalogs: true
I20260621 05:12:54.781198 16359 catalog_manager.cc:5782] T 00000000000000000000000000000000 P caf4f1ccbc684b5ca79d6a100cec1e37: Sending DeleteTablet for 3 replicas of tablet 2adae2fa6e294bf78ee0a8ba20cf8911
I20260621 05:12:54.781709 16613 tablet_service.cc:1514] Processing DeleteTablet for tablet 2adae2fa6e294bf78ee0a8ba20cf8911 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 05:12:54 UTC) from {username='kudu', principal='kudu/127.3.131.190@KRBTEST.COM'} at 127.0.0.1:36276
I20260621 05:12:54.781853 16760 tablet_service.cc:1514] Processing DeleteTablet for tablet 2adae2fa6e294bf78ee0a8ba20cf8911 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 05:12:54 UTC) from {username='kudu', principal='kudu/127.3.131.190@KRBTEST.COM'} at 127.0.0.1:35568
I20260621 05:12:54.782328 16492 tablet_service.cc:1514] Processing DeleteTablet for tablet 2adae2fa6e294bf78ee0a8ba20cf8911 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 05:12:54 UTC) from {username='kudu', principal='kudu/127.3.131.190@KRBTEST.COM'} at 127.0.0.1:45110
I20260621 05:12:54.782727 3598 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWZOx2_/build/release/bin/kudu with pid 16399
I20260621 05:12:54.784682 16884 tablet_replica.cc:330] T 2adae2fa6e294bf78ee0a8ba20cf8911 P 4644e1517d854a16a6716ea38148e104: stopping tablet replica
I20260621 05:12:54.784833 16884 raft_consensus.cc:2229] T 2adae2fa6e294bf78ee0a8ba20cf8911 P 4644e1517d854a16a6716ea38148e104 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260621 05:12:54.784976 16884 raft_consensus.cc:2258] T 2adae2fa6e294bf78ee0a8ba20cf8911 P 4644e1517d854a16a6716ea38148e104 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 05:12:54.785223 16883 tablet_replica.cc:330] T 2adae2fa6e294bf78ee0a8ba20cf8911 P a3fbf94490ec4c3d8e35edf5aaaef40a: stopping tablet replica
I20260621 05:12:54.785391 16883 raft_consensus.cc:2229] T 2adae2fa6e294bf78ee0a8ba20cf8911 P a3fbf94490ec4c3d8e35edf5aaaef40a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260621 05:12:54.785535 16883 raft_consensus.cc:2258] T 2adae2fa6e294bf78ee0a8ba20cf8911 P a3fbf94490ec4c3d8e35edf5aaaef40a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 05:12:54.785529 16884 ts_tablet_manager.cc:1905] T 2adae2fa6e294bf78ee0a8ba20cf8911 P 4644e1517d854a16a6716ea38148e104: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 05:12:54.786057 16883 ts_tablet_manager.cc:1905] T 2adae2fa6e294bf78ee0a8ba20cf8911 P a3fbf94490ec4c3d8e35edf5aaaef40a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 05:12:54.787356 16884 ts_tablet_manager.cc:1918] T 2adae2fa6e294bf78ee0a8ba20cf8911 P 4644e1517d854a16a6716ea38148e104: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260621 05:12:54.787505 16884 log.cc:1196] T 2adae2fa6e294bf78ee0a8ba20cf8911 P 4644e1517d854a16a6716ea38148e104: Deleting WAL directory at /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782018616648578-3598-0/minicluster-data/ts-2/wal/wals/2adae2fa6e294bf78ee0a8ba20cf8911
I20260621 05:12:54.787686 16883 ts_tablet_manager.cc:1918] T 2adae2fa6e294bf78ee0a8ba20cf8911 P a3fbf94490ec4c3d8e35edf5aaaef40a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260621 05:12:54.787835 16883 log.cc:1196] T 2adae2fa6e294bf78ee0a8ba20cf8911 P a3fbf94490ec4c3d8e35edf5aaaef40a: Deleting WAL directory at /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782018616648578-3598-0/minicluster-data/ts-1/wal/wals/2adae2fa6e294bf78ee0a8ba20cf8911
I20260621 05:12:54.787901 16884 ts_tablet_manager.cc:1939] T 2adae2fa6e294bf78ee0a8ba20cf8911 P 4644e1517d854a16a6716ea38148e104: Deleting consensus metadata
I20260621 05:12:54.788178 16883 ts_tablet_manager.cc:1939] T 2adae2fa6e294bf78ee0a8ba20cf8911 P a3fbf94490ec4c3d8e35edf5aaaef40a: Deleting consensus metadata
I20260621 05:12:54.788414 16346 catalog_manager.cc:4841] TS 4644e1517d854a16a6716ea38148e104 (127.3.131.131:33035): tablet 2adae2fa6e294bf78ee0a8ba20cf8911 (table test-table [id=b1c7d5b7fab349acb49b62be3993b94f]) successfully deleted
I20260621 05:12:54.788611 16344 catalog_manager.cc:4841] TS a3fbf94490ec4c3d8e35edf5aaaef40a (127.3.131.130:45271): tablet 2adae2fa6e294bf78ee0a8ba20cf8911 (table test-table [id=b1c7d5b7fab349acb49b62be3993b94f]) successfully deleted
W20260621 05:12:54.790547 16345 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.3.131.129:43057 (error 108)
I20260621 05:12:54.790902 3598 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWZOx2_/build/release/bin/kudu with pid 16560
I20260621 05:12:54.794754 16853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:12:54.794597 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.131.129:43057 (local address 127.0.0.1:45126)
0621 05:12:54.794652 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0621 05:12:54.794710 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.131.129:43057: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":22}
W20260621 05:12:54.795020 16345 catalog_manager.cc:4584] TS cddc12c2e002400ea87d54d68d4ceec6 (127.3.131.129:43057): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 2adae2fa6e294bf78ee0a8ba20cf8911: Network error: Client connection negotiation failed: client connection to 127.3.131.129:43057: connect: Connection refused (error 111)
I20260621 05:12:54.798714 3598 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWZOx2_/build/release/bin/kudu with pid 16695
I20260621 05:12:54.806553 3598 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWZOx2_/build/release/bin/kudu with pid 16327
I20260621 05:12:54.815165 3598 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260621 05:12:55.150393 3598 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260621 05:12:55.150480 3598 mini_ranger.cc:66] Stopping Ranger...
I20260621 05:12:55.495762 3598 mini_ranger.cc:68] Stopped Ranger
2026-06-21 05:12:55.495 UTC [15394] LOG: received smart shutdown request
2026-06-21 05:12:55.499 UTC [15394] LOG: background worker "logical replication launcher" (PID 15403) exited with exit code 1
2026-06-21 05:12:55.500 UTC [15398] LOG: shutting down
2026-06-21 05:12:55.701 UTC [15394] LOG: database system is shut down
2026-06-21T05:12:55Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (51674 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782018616648578-3598-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 21 05:12:55 dist-test-slave-vrq1 krb5kdc[16895](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 21 05:12:55 dist-test-slave-vrq1 krb5kdc[16895](info): set up 2 sockets
Jun 21 05:12:55 dist-test-slave-vrq1 krb5kdc[16895](info): commencing operation
krb5kdc: starting...
W20260621 05:12:56.867388 3598 mini_kdc.cc:121] Time spent starting KDC: real 1.027s user 0.006s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782018616648578-3598-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782018616648578-3598-0/krb5kdc/test-user.keytab.
Jun 21 05:12:56 dist-test-slave-vrq1 krb5kdc[16895](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782018776, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-21T05:12:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-21T05:12:56Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.131.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.131.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.131.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782018616648578-3598-0/krb5kdc/kudu_127.3.131.190.keytab.
Entry for principal kudu/127.3.131.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782018616648578-3598-0/krb5kdc/kudu_127.3.131.190.keytab.
I20260621 05:12:57.003811 3598 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWZOx2_/build/release/bin/kudu
/tmp/dist-test-taskWZOx2_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782018616648578-3598-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782018616648578-3598-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782018616648578-3598-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782018616648578-3598-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.131.190:41507
--webserver_interface=127.3.131.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782018616648578-3598-0/krb5kdc/kudu_127.3.131.190.keytab
--principal=kudu/127.3.131.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.131.148:33219
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.131.190:41507,127.3.131.189:37745,127.3.131.188:41779
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782018616648578-3598-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782018616648578-3598-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782018616648578-3598-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 05:12:57.092142 16908 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 05:12:57.092453 16908 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 05:12:57.092497 16908 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 05:12:57.093806 16908 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 05:12:57.093896 16908 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 05:12:57.093922 16908 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 05:12:57.093945 16908 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 05:12:57.093976 16908 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 05:12:57.095762 16908 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.131.148:33219
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782018616648578-3598-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782018616648578-3598-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.131.190:41507,127.3.131.189:37745,127.3.131.188:41779
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782018616648578-3598-0/krb5kdc/kudu_127.3.131.190.keytab
--principal=kudu/127.3.131.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.131.190:41507
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782018616648578-3598-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.131.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16908
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782018616648578-3598-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 21 Jun 2026 04:16:16 UTC on ccycloud.quasar-xycaha.root.comops.site
build id 4457881
I20260621 05:12:57.096093 16908 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 05:12:57.096354 16908 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 05:12:57.101671 16913 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:12:57.101671 16914 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:12:57.101691 16916 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:12:57.101990 16908 server_base.cc:1034] running on GCE node
I20260621 05:12:57.102257 16908 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 05:12:57.102573 16908 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 05:12:57.103747 16908 hybrid_clock.cc:648] HybridClock initialized: now 1782018777103712 us; error 53 us; skew 500 ppm
Jun 21 05:12:57 dist-test-slave-vrq1 krb5kdc[16895](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782018777, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.131.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 05:12:57.112449 16908 init.cc:377] Logged in from keytab as kudu/127.3.131.190@KRBTEST.COM (short username kudu)
I20260621 05:12:57.113111 16908 webserver.cc:460] Webserver started at http://127.3.131.190:40485/ using document root <none> and password file <none>
I20260621 05:12:57.113368 16908 fs_manager.cc:359] Metadata directory not provided
I20260621 05:12:57.113442 16908 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 05:12:57.113597 16908 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 05:12:57.114713 16908 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782018616648578-3598-0/minicluster-data/master-0/data/instance:
uuid: "7b5dd37dd079413f8c0ab35ffb1451e9"
format_stamp: "Formatted at 2026-06-21 05:12:57 on dist-test-slave-vrq1"
server_key: "89212840ce4d89988260664094d3a756"
server_key_iv: "d5cddb14e0160363943a1eb042a54467"
server_key_version: "encryptionkey@0"
I20260621 05:12:57.115105 16908 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782018616648578-3598-0/minicluster-data/master-0/wal/instance:
uuid: "7b5dd37dd079413f8c0ab35ffb1451e9"
format_stamp: "Formatted at 2026-06-21 05:12:57 on dist-test-slave-vrq1"
server_key: "89212840ce4d89988260664094d3a756"
server_key_iv: "d5cddb14e0160363943a1eb042a54467"
server_key_version: "encryptionkey@0"
I20260621 05:12:57.116642 16908 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260621 05:12:57.117590 16923 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 05:12:57.117834 16908 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260621 05:12:57.117956 16908 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782018616648578-3598-0/minicluster-data/master-0/data,/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782018616648578-3598-0/minicluster-data/master-0/wal
uuid: "7b5dd37dd079413f8c0ab35ffb1451e9"
format_stamp: "Formatted at 2026-06-21 05:12:57 on dist-test-slave-vrq1"
server_key: "89212840ce4d89988260664094d3a756"
server_key_iv: "d5cddb14e0160363943a1eb042a54467"
server_key_version: "encryptionkey@0"
I20260621 05:12:57.118042 16908 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782018616648578-3598-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782018616648578-3598-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782018616648578-3598-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 05:12:57.132863 16908 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 05:12:57.135591 16908 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 05:12:57.135788 16908 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 05:12:57.140446 16908 rpc_server.cc:307] RPC server started. Bound to: 127.3.131.190:41507
I20260621 05:12:57.140486 16975 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.131.190:41507 every 8 connection(s)
I20260621 05:12:57.140872 16908 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782018616648578-3598-0/minicluster-data/master-0/data/info.pb
I20260621 05:12:57.141772 16976 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 05:12:57.143479 16976 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.131.190" port: 41507 } has no permanent_uuid. Determining permanent_uuid...
I20260621 05:12:57.149598 3598 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWZOx2_/build/release/bin/kudu as pid 16908
I20260621 05:12:57.149686 3598 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782018616648578-3598-0/minicluster-data/master-0/wal/instance
I20260621 05:12:57.149904 3598 external_mini_cluster.cc:1428] Setting key a30b026ae467a3b2a84a4c6abef98d7c
Jun 21 05:12:57 dist-test-slave-vrq1 krb5kdc[16895](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782018777, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.131.190@KRBTEST.COM for kudu/127.3.131.190@KRBTEST.COM
I20260621 05:12:57.154644 16979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:12:57.144320 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59490 (local address 127.3.131.190:41507)
0621 05:12:57.144545 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0621 05:12:57.144555 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0621 05:12:57.144688 (+ 133us) server_negotiation.cc:407] Connection header received
0621 05:12:57.145312 (+ 624us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:12:57.145314 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:12:57.145340 (+ 26us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:12:57.145445 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:12:57.146038 (+ 593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:12:57.146854 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:12:57.147801 (+ 947us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:12:57.148117 (+ 316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:12:57.148244 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:12:57.150648 (+ 2404us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:12:57.150657 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:12:57.150657 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:12:57.150667 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:12:57.152682 (+ 2015us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:12:57.153225 (+ 543us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:12:57.153228 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:12:57.153229 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:12:57.153325 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:12:57.153650 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:12:57.153651 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:12:57.153652 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:12:57.153930 (+ 278us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:12:57.154082 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0621 05:12:57.154334 (+ 252us) server_negotiation.cc:299] Negotiation successful
0621 05:12:57.154449 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":176,"spinlock_wait_cycles":8960,"thread_start_us":115,"threads_started":1}
I20260621 05:12:57.154847 16978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:12:57.144289 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.131.190:41507 (local address 127.0.0.1:59490)
0621 05:12:57.144534 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0621 05:12:57.144581 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0621 05:12:57.145144 (+ 563us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:12:57.145452 (+ 308us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:12:57.145455 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:12:57.145757 (+ 302us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:12:57.145927 (+ 170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:12:57.145931 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:12:57.146955 (+ 1024us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:12:57.146956 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:12:57.147706 (+ 750us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:12:57.147709 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:12:57.148213 (+ 504us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:12:57.148214 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:12:57.148287 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:12:57.148897 (+ 610us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 05:12:57.148906 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:12:57.150554 (+ 1648us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 05:12:57.152794 (+ 2240us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:12:57.152803 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:12:57.152806 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:12:57.153111 (+ 305us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:12:57.153410 (+ 299us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:12:57.153411 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:12:57.153411 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:12:57.153533 (+ 122us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:12:57.154063 (+ 530us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 05:12:57.154071 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 05:12:57.154162 (+ 91us) client_negotiation.cc:769] Sending connection context
0621 05:12:57.154279 (+ 117us) client_negotiation.cc:240] Negotiation successful
0621 05:12:57.154564 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":188,"mutex_wait_us":90,"thread_start_us":101,"threads_started":1}
I20260621 05:12:57.155642 16976 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.131.189" port: 37745 } has no permanent_uuid. Determining permanent_uuid...
I20260621 05:12:57.156169 16978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:12:57.156019 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.131.189:37745 (local address 127.0.0.1:35972)
0621 05:12:57.156094 (+ 75us) negotiation.cc:107] Waiting for socket to connect
0621 05:12:57.156144 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.131.189:37745: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260621 05:12:57.156399 16924 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.131.189:37745: connect: Connection refused (error 111)
I20260621 05:12:57.156982 16978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:12:57.156839 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.131.189:37745 (local address 127.0.0.1:35988)
0621 05:12:57.156930 (+ 91us) negotiation.cc:107] Waiting for socket to connect
0621 05:12:57.156961 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.131.189:37745: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
W20260621 05:12:57.157133 16976 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.131.189:37745: Network error: Client connection negotiation failed: client connection to 127.3.131.189:37745: connect: Connection refused (error 111)
WARNING: no policy specified for kudu/127.3.131.189@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.131.189@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.131.189 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782018616648578-3598-0/krb5kdc/kudu_127.3.131.189.keytab.
Entry for principal kudu/127.3.131.189 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782018616648578-3598-0/krb5kdc/kudu_127.3.131.189.keytab.
I20260621 05:12:57.178939 3598 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWZOx2_/build/release/bin/kudu
/tmp/dist-test-taskWZOx2_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782018616648578-3598-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782018616648578-3598-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782018616648578-3598-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782018616648578-3598-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.131.189:37745
--webserver_interface=127.3.131.189
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782018616648578-3598-0/krb5kdc/kudu_127.3.131.189.keytab
--principal=kudu/127.3.131.189
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.131.148:33219
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.131.190:41507,127.3.131.189:37745,127.3.131.188:41779
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782018616648578-3598-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782018616648578-3598-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782018616648578-3598-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260621 05:12:57.206399 16976 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.131.189" port: 37745 } attempt: 1
I20260621 05:12:57.206976 16978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:12:57.206759 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.131.189:37745 (local address 127.0.0.1:35994)
0621 05:12:57.206901 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0621 05:12:57.206945 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.131.189:37745: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":119}
I20260621 05:12:57.207357 16978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:12:57.207258 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.131.189:37745 (local address 127.0.0.1:36010)
0621 05:12:57.207302 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0621 05:12:57.207331 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.131.189:37745: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260621 05:12:57.207463 16976 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.131.189:37745: Network error: Client connection negotiation failed: client connection to 127.3.131.189:37745: connect: Connection refused (error 111)
W20260621 05:12:57.260947 16983 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 05:12:57.261229 16983 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 05:12:57.261266 16983 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 05:12:57.262494 16983 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 05:12:57.262575 16983 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 05:12:57.262599 16983 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 05:12:57.262620 16983 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 05:12:57.262652 16983 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 05:12:57.264277 16983 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.131.148:33219
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782018616648578-3598-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782018616648578-3598-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.3.131.190:41507,127.3.131.189:37745,127.3.131.188:41779
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782018616648578-3598-0/krb5kdc/kudu_127.3.131.189.keytab
--principal=kudu/127.3.131.189
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.131.189:37745
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782018616648578-3598-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.3.131.189
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16983
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782018616648578-3598-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 21 Jun 2026 04:16:16 UTC on ccycloud.quasar-xycaha.root.comops.site
build id 4457881
I20260621 05:12:57.264580 16983 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 05:12:57.264817 16983 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260621 05:12:57.270179 16983 server_base.cc:1034] running on GCE node
W20260621 05:12:57.270218 16989 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:12:57.270219 16991 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:12:57.270334 16988 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:12:57.270761 16983 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 05:12:57.271098 16983 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 05:12:57.272284 16983 hybrid_clock.cc:648] HybridClock initialized: now 1782018777272333 us; error 106 us; skew 500 ppm
Jun 21 05:12:57 dist-test-slave-vrq1 krb5kdc[16895](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782018777, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.131.189@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 05:12:57.275084 16983 init.cc:377] Logged in from keytab as kudu/127.3.131.189@KRBTEST.COM (short username kudu)
I20260621 05:12:57.275668 16983 webserver.cc:460] Webserver started at http://127.3.131.189:41393/ using document root <none> and password file <none>
I20260621 05:12:57.275620 16976 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.131.189" port: 37745 } attempt: 2
I20260621 05:12:57.275975 16983 fs_manager.cc:359] Metadata directory not provided
I20260621 05:12:57.276051 16983 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 05:12:57.276186 16978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:12:57.275992 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.131.189:37745 (local address 127.0.0.1:36014)
0621 05:12:57.276099 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0621 05:12:57.276142 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.131.189:37745: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
I20260621 05:12:57.276201 16983 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 05:12:57.276679 16978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:12:57.276550 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.131.189:37745 (local address 127.0.0.1:36028)
0621 05:12:57.276622 (+ 72us) negotiation.cc:107] Waiting for socket to connect
0621 05:12:57.276654 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.131.189:37745: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260621 05:12:57.276850 16976 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.131.189:37745: Network error: Client connection negotiation failed: client connection to 127.3.131.189:37745: connect: Connection refused (error 111)
I20260621 05:12:57.277540 16983 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782018616648578-3598-0/minicluster-data/master-1/data/instance:
uuid: "7c7ba507fa614e6396d258655e2f59d8"
format_stamp: "Formatted at 2026-06-21 05:12:57 on dist-test-slave-vrq1"
server_key: "a0dae6cf757d805099007613c85e811c"
server_key_iv: "0aa5d1215f33caab00c666aeb238f24e"
server_key_version: "encryptionkey@0"
I20260621 05:12:57.277904 16983 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782018616648578-3598-0/minicluster-data/master-1/wal/instance:
uuid: "7c7ba507fa614e6396d258655e2f59d8"
format_stamp: "Formatted at 2026-06-21 05:12:57 on dist-test-slave-vrq1"
server_key: "a0dae6cf757d805099007613c85e811c"
server_key_iv: "0aa5d1215f33caab00c666aeb238f24e"
server_key_version: "encryptionkey@0"
I20260621 05:12:57.279297 16983 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260621 05:12:57.280128 16998 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 05:12:57.280321 16983 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260621 05:12:57.280453 16983 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782018616648578-3598-0/minicluster-data/master-1/data,/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782018616648578-3598-0/minicluster-data/master-1/wal
uuid: "7c7ba507fa614e6396d258655e2f59d8"
format_stamp: "Formatted at 2026-06-21 05:12:57 on dist-test-slave-vrq1"
server_key: "a0dae6cf757d805099007613c85e811c"
server_key_iv: "0aa5d1215f33caab00c666aeb238f24e"
server_key_version: "encryptionkey@0"
I20260621 05:12:57.280524 16983 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782018616648578-3598-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782018616648578-3598-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782018616648578-3598-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 05:12:57.290572 16983 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 05:12:57.292955 16983 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 05:12:57.293129 16983 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 05:12:57.298547 16983 rpc_server.cc:307] RPC server started. Bound to: 127.3.131.189:37745
I20260621 05:12:57.298609 17050 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.131.189:37745 every 8 connection(s)
I20260621 05:12:57.299067 16983 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782018616648578-3598-0/minicluster-data/master-1/data/info.pb
I20260621 05:12:57.300239 17051 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 05:12:57.302356 17051 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.131.190" port: 41507 } has no permanent_uuid. Determining permanent_uuid...
I20260621 05:12:57.304190 3598 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWZOx2_/build/release/bin/kudu as pid 16983
I20260621 05:12:57.304313 3598 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782018616648578-3598-0/minicluster-data/master-1/wal/instance
I20260621 05:12:57.304512 3598 external_mini_cluster.cc:1428] Setting key 8af0cce55f57aa7ab32a5c39e274ab36
Jun 21 05:12:57 dist-test-slave-vrq1 krb5kdc[16895](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782018777, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.131.189@KRBTEST.COM for kudu/127.3.131.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.3.131.188@KRBTEST.COM; defaulting to no policy
I20260621 05:12:57.313786 16979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:12:57.303290 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59496 (local address 127.3.131.190:41507)
0621 05:12:57.303343 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0621 05:12:57.303345 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:12:57.304392 (+ 1047us) server_negotiation.cc:407] Connection header received
0621 05:12:57.305115 (+ 723us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:12:57.305117 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:12:57.305134 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:12:57.305193 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:12:57.305880 (+ 687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:12:57.306391 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:12:57.307476 (+ 1085us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:12:57.307806 (+ 330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:12:57.307878 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:12:57.310366 (+ 2488us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:12:57.310368 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:12:57.310368 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:12:57.310378 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:12:57.312130 (+ 1752us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:12:57.312640 (+ 510us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:12:57.312641 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:12:57.312641 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:12:57.312702 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:12:57.313056 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:12:57.313057 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:12:57.313057 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:12:57.313202 (+ 145us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:12:57.313281 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0621 05:12:57.313618 (+ 337us) server_negotiation.cc:299] Negotiation successful
0621 05:12:57.313711 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260621 05:12:57.313980 17053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:12:57.303281 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.131.190:41507 (local address 127.0.0.1:59496)
0621 05:12:57.304240 (+ 959us) negotiation.cc:107] Waiting for socket to connect
0621 05:12:57.304295 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0621 05:12:57.304880 (+ 585us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:12:57.305279 (+ 399us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:12:57.305284 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:12:57.305555 (+ 271us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:12:57.305772 (+ 217us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:12:57.305778 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:12:57.306493 (+ 715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:12:57.306494 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:12:57.307387 (+ 893us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:12:57.307390 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:12:57.307914 (+ 524us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:12:57.307915 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:12:57.308050 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:12:57.308663 (+ 613us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 05:12:57.308672 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:12:57.310259 (+ 1587us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 05:12:57.312252 (+ 1993us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:12:57.312259 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:12:57.312261 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:12:57.312548 (+ 287us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:12:57.312843 (+ 295us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:12:57.312844 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:12:57.312844 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:12:57.312975 (+ 131us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:12:57.313277 (+ 302us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 05:12:57.313282 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 05:12:57.313483 (+ 201us) client_negotiation.cc:769] Sending connection context
0621 05:12:57.313574 (+ 91us) client_negotiation.cc:240] Negotiation successful
0621 05:12:57.313712 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":903,"thread_start_us":79,"threads_started":1}
I20260621 05:12:57.314590 17051 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.131.189" port: 37745 } has no permanent_uuid. Determining permanent_uuid...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.131.188@KRBTEST.COM" created.
Jun 21 05:12:57 dist-test-slave-vrq1 krb5kdc[16895](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782018777, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.131.189@KRBTEST.COM for kudu/127.3.131.189@KRBTEST.COM
I20260621 05:12:57.323545 17053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:12:57.314869 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.131.189:37745 (local address 127.0.0.1:36030)
0621 05:12:57.314921 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0621 05:12:57.314933 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0621 05:12:57.314974 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:12:57.315439 (+ 465us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:12:57.315440 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:12:57.315561 (+ 121us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:12:57.315626 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:12:57.315628 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:12:57.316875 (+ 1247us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:12:57.316877 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:12:57.317732 (+ 855us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:12:57.317735 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:12:57.318233 (+ 498us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:12:57.318234 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:12:57.318324 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:12:57.318765 (+ 441us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 05:12:57.318770 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:12:57.320406 (+ 1636us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 05:12:57.322428 (+ 2022us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:12:57.322429 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:12:57.322429 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:12:57.322603 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:12:57.322816 (+ 213us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:12:57.322816 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:12:57.322816 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:12:57.322870 (+ 54us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:12:57.323156 (+ 286us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 05:12:57.323157 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 05:12:57.323237 (+ 80us) client_negotiation.cc:769] Sending connection context
0621 05:12:57.323323 (+ 86us) client_negotiation.cc:240] Negotiation successful
0621 05:12:57.323427 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33}
I20260621 05:12:57.323614 17055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:12:57.314957 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36030 (local address 127.3.131.189:37745)
0621 05:12:57.315144 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0621 05:12:57.315148 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 05:12:57.315165 (+ 17us) server_negotiation.cc:407] Connection header received
0621 05:12:57.315257 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:12:57.315260 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:12:57.315311 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:12:57.315420 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:12:57.315719 (+ 299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:12:57.316762 (+ 1043us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:12:57.317848 (+ 1086us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:12:57.318146 (+ 298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:12:57.318214 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:12:57.320505 (+ 2291us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:12:57.320514 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:12:57.320514 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:12:57.320522 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:12:57.322335 (+ 1813us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:12:57.322696 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:12:57.322701 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:12:57.322705 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:12:57.322751 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:12:57.322940 (+ 189us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:12:57.322941 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:12:57.322941 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:12:57.323081 (+ 140us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:12:57.323168 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0621 05:12:57.323355 (+ 187us) server_negotiation.cc:299] Negotiation successful
0621 05:12:57.323480 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":60,"server-negotiator.queue_time_us":159,"thread_start_us":111,"threads_started":1}
I20260621 05:12:57.324182 17051 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.131.188" port: 41779 } has no permanent_uuid. Determining permanent_uuid...
I20260621 05:12:57.324540 17053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:12:57.324420 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.131.188:41779 (local address 127.0.0.1:57674)
0621 05:12:57.324468 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0621 05:12:57.324512 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.131.188:41779: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260621 05:12:57.324661 16999 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.131.188:41779: connect: Connection refused (error 111)
I20260621 05:12:57.325069 17053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:12:57.324974 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.131.188:41779 (local address 127.0.0.1:57676)
0621 05:12:57.325024 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0621 05:12:57.325047 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.131.188:41779: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260621 05:12:57.325176 17051 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.131.188:41779: Network error: Client connection negotiation failed: client connection to 127.3.131.188:41779: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.131.188 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782018616648578-3598-0/krb5kdc/kudu_127.3.131.188.keytab.
Entry for principal kudu/127.3.131.188 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782018616648578-3598-0/krb5kdc/kudu_127.3.131.188.keytab.
I20260621 05:12:57.334352 3598 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWZOx2_/build/release/bin/kudu
/tmp/dist-test-taskWZOx2_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782018616648578-3598-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782018616648578-3598-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782018616648578-3598-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782018616648578-3598-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.131.188:41779
--webserver_interface=127.3.131.188
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782018616648578-3598-0/krb5kdc/kudu_127.3.131.188.keytab
--principal=kudu/127.3.131.188
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.131.148:33219
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.131.190:41507,127.3.131.189:37745,127.3.131.188:41779
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782018616648578-3598-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782018616648578-3598-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782018616648578-3598-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260621 05:12:57.368021 16976 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.131.189" port: 37745 } attempt: 3
Jun 21 05:12:57 dist-test-slave-vrq1 krb5kdc[16895](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782018777, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.131.190@KRBTEST.COM for kudu/127.3.131.189@KRBTEST.COM
I20260621 05:12:57.374413 17051 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.131.188" port: 41779 } attempt: 1
I20260621 05:12:57.374852 17053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:12:57.374696 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.131.188:41779 (local address 127.0.0.1:57690)
0621 05:12:57.374767 (+ 71us) negotiation.cc:107] Waiting for socket to connect
0621 05:12:57.374807 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.131.188:41779: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
I20260621 05:12:57.375309 17053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:12:57.375228 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.131.188:41779 (local address 127.0.0.1:57702)
0621 05:12:57.375264 (+ 36us) negotiation.cc:107] Waiting for socket to connect
0621 05:12:57.375289 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.131.188:41779: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":16}
W20260621 05:12:57.375571 17051 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.131.188:41779: Network error: Client connection negotiation failed: client connection to 127.3.131.188:41779: connect: Connection refused (error 111)
I20260621 05:12:57.375856 17055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:12:57.368557 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36032 (local address 127.3.131.189:37745)
0621 05:12:57.368621 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0621 05:12:57.368624 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0621 05:12:57.368637 (+ 13us) server_negotiation.cc:407] Connection header received
0621 05:12:57.368721 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:12:57.368722 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:12:57.368738 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:12:57.368811 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:12:57.369160 (+ 349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:12:57.369670 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:12:57.370363 (+ 693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:12:57.370576 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:12:57.370656 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:12:57.372742 (+ 2086us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:12:57.372743 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:12:57.372744 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:12:57.372751 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:12:57.374418 (+ 1667us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:12:57.374872 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:12:57.374873 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:12:57.374874 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:12:57.374942 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:12:57.375270 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:12:57.375271 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:12:57.375271 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:12:57.375440 (+ 169us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:12:57.375509 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0621 05:12:57.375693 (+ 184us) server_negotiation.cc:299] Negotiation successful
0621 05:12:57.375777 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260621 05:12:57.375867 16978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:12:57.368499 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.131.189:37745 (local address 127.0.0.1:36032)
0621 05:12:57.368587 (+ 88us) negotiation.cc:107] Waiting for socket to connect
0621 05:12:57.368601 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0621 05:12:57.368653 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:12:57.368820 (+ 167us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:12:57.368821 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:12:57.369006 (+ 185us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:12:57.369072 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:12:57.369074 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:12:57.369763 (+ 689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:12:57.369764 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:12:57.370280 (+ 516us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:12:57.370282 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:12:57.370652 (+ 370us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:12:57.370653 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:12:57.370738 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:12:57.371253 (+ 515us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 05:12:57.371262 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:12:57.372620 (+ 1358us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 05:12:57.374531 (+ 1911us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:12:57.374532 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:12:57.374533 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:12:57.374789 (+ 256us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:12:57.375008 (+ 219us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:12:57.375009 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:12:57.375009 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:12:57.375082 (+ 73us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:12:57.375511 (+ 429us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 05:12:57.375512 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 05:12:57.375585 (+ 73us) client_negotiation.cc:769] Sending connection context
0621 05:12:57.375665 (+ 80us) client_negotiation.cc:240] Negotiation successful
0621 05:12:57.375773 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":54}
I20260621 05:12:57.376585 16976 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.131.188" port: 41779 } has no permanent_uuid. Determining permanent_uuid...
I20260621 05:12:57.376911 16978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:12:57.376807 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.131.188:41779 (local address 127.0.0.1:57718)
0621 05:12:57.376855 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0621 05:12:57.376891 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.131.188:41779: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
I20260621 05:12:57.377177 16978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:12:57.377126 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.131.188:41779 (local address 127.0.0.1:57728)
0621 05:12:57.377146 (+ 20us) negotiation.cc:107] Waiting for socket to connect
0621 05:12:57.377165 (+ 19us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.131.188:41779: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":13}
W20260621 05:12:57.377272 16976 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.131.188:41779: Network error: Client connection negotiation failed: client connection to 127.3.131.188:41779: connect: Connection refused (error 111)
I20260621 05:12:57.408447 16976 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.131.188" port: 41779 } attempt: 1
I20260621 05:12:57.409046 16978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:12:57.408898 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.131.188:41779 (local address 127.0.0.1:57742)
0621 05:12:57.408972 (+ 74us) negotiation.cc:107] Waiting for socket to connect
0621 05:12:57.409016 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.131.188:41779: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
I20260621 05:12:57.409402 16978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:12:57.409296 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.131.188:41779 (local address 127.0.0.1:57748)
0621 05:12:57.409346 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0621 05:12:57.409381 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.131.188:41779: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260621 05:12:57.409507 16976 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.131.188:41779: Network error: Client connection negotiation failed: client connection to 127.3.131.188:41779: connect: Connection refused (error 111)
W20260621 05:12:57.419458 17058 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 05:12:57.419731 17058 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 05:12:57.419773 17058 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 05:12:57.421080 17058 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 05:12:57.421159 17058 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 05:12:57.421181 17058 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 05:12:57.421203 17058 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 05:12:57.421233 17058 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 05:12:57.422899 17058 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.131.148:33219
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782018616648578-3598-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782018616648578-3598-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.3.131.190:41507,127.3.131.189:37745,127.3.131.188:41779
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782018616648578-3598-0/krb5kdc/kudu_127.3.131.188.keytab
--principal=kudu/127.3.131.188
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.131.188:41779
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782018616648578-3598-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.3.131.188
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17058
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782018616648578-3598-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 21 Jun 2026 04:16:16 UTC on ccycloud.quasar-xycaha.root.comops.site
build id 4457881
I20260621 05:12:57.423229 17058 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 05:12:57.423487 17058 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 05:12:57.428922 17063 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:12:57.429018 17058 server_base.cc:1034] running on GCE node
W20260621 05:12:57.428992 17066 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:12:57.429111 17064 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:12:57.429373 17058 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 05:12:57.429647 17058 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 05:12:57.430862 17058 hybrid_clock.cc:648] HybridClock initialized: now 1782018777430822 us; error 53 us; skew 500 ppm
Jun 21 05:12:57 dist-test-slave-vrq1 krb5kdc[16895](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782018777, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.131.188@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 05:12:57.433531 17058 init.cc:377] Logged in from keytab as kudu/127.3.131.188@KRBTEST.COM (short username kudu)
I20260621 05:12:57.433933 17058 webserver.cc:460] Webserver started at http://127.3.131.188:38435/ using document root <none> and password file <none>
I20260621 05:12:57.434182 17058 fs_manager.cc:359] Metadata directory not provided
I20260621 05:12:57.434229 17058 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 05:12:57.434369 17058 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 05:12:57.435545 17058 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782018616648578-3598-0/minicluster-data/master-2/data/instance:
uuid: "8d1f2a82c8194921b40e47d32864c72d"
format_stamp: "Formatted at 2026-06-21 05:12:57 on dist-test-slave-vrq1"
server_key: "38eb491ef3ec1a9d044d6c362207893a"
server_key_iv: "4b087edb67fd160f61916a225a2a6d32"
server_key_version: "encryptionkey@0"
I20260621 05:12:57.435914 17058 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782018616648578-3598-0/minicluster-data/master-2/wal/instance:
uuid: "8d1f2a82c8194921b40e47d32864c72d"
format_stamp: "Formatted at 2026-06-21 05:12:57 on dist-test-slave-vrq1"
server_key: "38eb491ef3ec1a9d044d6c362207893a"
server_key_iv: "4b087edb67fd160f61916a225a2a6d32"
server_key_version: "encryptionkey@0"
I20260621 05:12:57.437268 17058 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.001s
I20260621 05:12:57.438060 17073 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 05:12:57.438239 17058 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260621 05:12:57.438339 17058 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782018616648578-3598-0/minicluster-data/master-2/data,/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782018616648578-3598-0/minicluster-data/master-2/wal
uuid: "8d1f2a82c8194921b40e47d32864c72d"
format_stamp: "Formatted at 2026-06-21 05:12:57 on dist-test-slave-vrq1"
server_key: "38eb491ef3ec1a9d044d6c362207893a"
server_key_iv: "4b087edb67fd160f61916a225a2a6d32"
server_key_version: "encryptionkey@0"
I20260621 05:12:57.438421 17058 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782018616648578-3598-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782018616648578-3598-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782018616648578-3598-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 05:12:57.443727 17051 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.131.188" port: 41779 } attempt: 2
I20260621 05:12:57.444249 17053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:12:57.444075 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.131.188:41779 (local address 127.0.0.1:57754)
0621 05:12:57.444152 (+ 77us) negotiation.cc:107] Waiting for socket to connect
0621 05:12:57.444218 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.131.188:41779: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260621 05:12:57.444630 17053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:12:57.444524 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.131.188:41779 (local address 127.0.0.1:57760)
0621 05:12:57.444574 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0621 05:12:57.444608 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.131.188:41779: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260621 05:12:57.444796 17051 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.131.188:41779: Network error: Client connection negotiation failed: client connection to 127.3.131.188:41779: connect: Connection refused (error 111)
I20260621 05:12:57.459990 17058 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 05:12:57.462502 17058 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 05:12:57.462669 17058 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 05:12:57.467058 17058 rpc_server.cc:307] RPC server started. Bound to: 127.3.131.188:41779
I20260621 05:12:57.467178 17125 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.131.188:41779 every 8 connection(s)
I20260621 05:12:57.467525 17058 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782018616648578-3598-0/minicluster-data/master-2/data/info.pb
I20260621 05:12:57.468456 17126 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 05:12:57.469563 3598 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWZOx2_/build/release/bin/kudu as pid 17058
I20260621 05:12:57.469650 3598 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782018616648578-3598-0/minicluster-data/master-2/wal/instance
I20260621 05:12:57.470027 3598 external_mini_cluster.cc:1428] Setting key 12c16334d9c630b72e67461c082da310
I20260621 05:12:57.470273 17126 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.131.190" port: 41507 } has no permanent_uuid. Determining permanent_uuid...
Jun 21 05:12:57 dist-test-slave-vrq1 krb5kdc[16895](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782018777, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.131.188@KRBTEST.COM for kudu/127.3.131.190@KRBTEST.COM
Jun 21 05:12:57 dist-test-slave-vrq1 krb5kdc[16895](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782018776, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.131.188@KRBTEST.COM
I20260621 05:12:57.480952 16979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:12:57.471099 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59498 (local address 127.3.131.190:41507)
0621 05:12:57.471148 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0621 05:12:57.471150 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:12:57.471345 (+ 195us) server_negotiation.cc:407] Connection header received
0621 05:12:57.471551 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:12:57.471552 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:12:57.471564 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:12:57.471637 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:12:57.472235 (+ 598us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:12:57.472726 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:12:57.473807 (+ 1081us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:12:57.474049 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:12:57.474133 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:12:57.477103 (+ 2970us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:12:57.477109 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:12:57.477110 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:12:57.477117 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:12:57.478918 (+ 1801us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:12:57.479599 (+ 681us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:12:57.479601 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:12:57.479602 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:12:57.479732 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:12:57.480254 (+ 522us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:12:57.480255 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:12:57.480256 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:12:57.480391 (+ 135us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:12:57.480465 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0621 05:12:57.480754 (+ 289us) server_negotiation.cc:299] Negotiation successful
0621 05:12:57.480855 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260621 05:12:57.481184 17130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:12:57.470978 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.131.190:41507 (local address 127.0.0.1:59498)
0621 05:12:57.471150 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0621 05:12:57.471226 (+ 76us) client_negotiation.cc:174] Beginning negotiation
0621 05:12:57.471468 (+ 242us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:12:57.471647 (+ 179us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:12:57.471652 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:12:57.471966 (+ 314us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:12:57.472141 (+ 175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:12:57.472145 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:12:57.472981 (+ 836us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:12:57.472982 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:12:57.473708 (+ 726us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:12:57.473710 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:12:57.474139 (+ 429us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:12:57.474140 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:12:57.474301 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:12:57.474889 (+ 588us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 05:12:57.474900 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:12:57.476986 (+ 2086us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 05:12:57.479033 (+ 2047us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:12:57.479041 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:12:57.479043 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:12:57.479486 (+ 443us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:12:57.480027 (+ 541us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:12:57.480029 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:12:57.480029 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:12:57.480171 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:12:57.480468 (+ 297us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 05:12:57.480475 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 05:12:57.480626 (+ 151us) client_negotiation.cc:769] Sending connection context
0621 05:12:57.480721 (+ 95us) client_negotiation.cc:240] Negotiation successful
0621 05:12:57.480888 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":137,"mutex_wait_us":1,"thread_start_us":72,"threads_started":1}
I20260621 05:12:57.481196 17129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:12:57.470617 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57762 (local address 127.3.131.188:41779)
0621 05:12:57.470816 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0621 05:12:57.470826 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0621 05:12:57.470911 (+ 85us) server_negotiation.cc:407] Connection header received
0621 05:12:57.471472 (+ 561us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:12:57.471481 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:12:57.471507 (+ 26us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:12:57.471771 (+ 264us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:12:57.472456 (+ 685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:12:57.473274 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:12:57.474041 (+ 767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:12:57.474421 (+ 380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:12:57.474506 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:12:57.477303 (+ 2797us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:12:57.477312 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:12:57.477312 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:12:57.477328 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:12:57.479577 (+ 2249us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:12:57.480118 (+ 541us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:12:57.480121 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:12:57.480122 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:12:57.480184 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:12:57.480435 (+ 251us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:12:57.480435 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:12:57.480436 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:12:57.480631 (+ 195us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:12:57.480794 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0621 05:12:57.480869 (+ 75us) server_negotiation.cc:299] Negotiation successful
0621 05:12:57.480956 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":65,"server-negotiator.queue_time_us":126,"thread_start_us":92,"threads_started":1}
I20260621 05:12:57.481710 17126 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.131.189" port: 37745 } has no permanent_uuid. Determining permanent_uuid...
I20260621 05:12:57.484655 16976 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.131.188" port: 41779 } attempt: 2
Jun 21 05:12:57 dist-test-slave-vrq1 krb5kdc[16895](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782018777, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.131.188@KRBTEST.COM for kudu/127.3.131.189@KRBTEST.COM
Jun 21 05:12:57 dist-test-slave-vrq1 krb5kdc[16895](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782018776, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.131.190@KRBTEST.COM
Jun 21 05:12:57 dist-test-slave-vrq1 krb5kdc[16895](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782018777, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.131.190@KRBTEST.COM for kudu/127.3.131.188@KRBTEST.COM
I20260621 05:12:57.490502 17055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:12:57.482150 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36042 (local address 127.3.131.189:37745)
0621 05:12:57.482195 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0621 05:12:57.482197 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:12:57.482207 (+ 10us) server_negotiation.cc:407] Connection header received
0621 05:12:57.482235 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:12:57.482236 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:12:57.482248 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:12:57.482426 (+ 178us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:12:57.482735 (+ 309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:12:57.483532 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:12:57.484454 (+ 922us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:12:57.484705 (+ 251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:12:57.484781 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:12:57.487432 (+ 2651us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:12:57.487434 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:12:57.487434 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:12:57.487444 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:12:57.489177 (+ 1733us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:12:57.489569 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:12:57.489570 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:12:57.489571 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:12:57.489639 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:12:57.489848 (+ 209us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:12:57.489849 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:12:57.489849 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:12:57.490019 (+ 170us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:12:57.490092 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0621 05:12:57.490297 (+ 205us) server_negotiation.cc:299] Negotiation successful
0621 05:12:57.490389 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260621 05:12:57.490854 17130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:12:57.482070 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.131.189:37745 (local address 127.0.0.1:36042)
0621 05:12:57.482136 (+ 66us) negotiation.cc:107] Waiting for socket to connect
0621 05:12:57.482147 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0621 05:12:57.482182 (+ 35us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:12:57.482423 (+ 241us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:12:57.482424 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:12:57.482596 (+ 172us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:12:57.482656 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:12:57.482657 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:12:57.483648 (+ 991us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:12:57.483649 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:12:57.484361 (+ 712us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:12:57.484364 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:12:57.484847 (+ 483us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:12:57.484848 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:12:57.484941 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:12:57.485485 (+ 544us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 05:12:57.485491 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:12:57.487324 (+ 1833us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 05:12:57.489280 (+ 1956us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:12:57.489281 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:12:57.489282 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:12:57.489470 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:12:57.489709 (+ 239us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:12:57.489710 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:12:57.489710 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:12:57.489778 (+ 68us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:12:57.490100 (+ 322us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 05:12:57.490101 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 05:12:57.490153 (+ 52us) client_negotiation.cc:769] Sending connection context
0621 05:12:57.490636 (+ 483us) client_negotiation.cc:240] Negotiation successful
0621 05:12:57.490763 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":29}
I20260621 05:12:57.491333 17126 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.131.188" port: 41779 } has no permanent_uuid. Determining permanent_uuid...
I20260621 05:12:57.491885 16979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:12:57.482223 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59508 (local address 127.3.131.190:41507)
0621 05:12:57.482287 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0621 05:12:57.482288 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0621 05:12:57.482416 (+ 128us) server_negotiation.cc:407] Connection header received
0621 05:12:57.482481 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:12:57.482482 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:12:57.482492 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:12:57.482554 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:12:57.483137 (+ 583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:12:57.484079 (+ 942us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:12:57.484968 (+ 889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:12:57.485243 (+ 275us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:12:57.485327 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:12:57.488089 (+ 2762us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:12:57.488090 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:12:57.488090 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:12:57.488098 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:12:57.489894 (+ 1796us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:12:57.490629 (+ 735us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:12:57.490630 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:12:57.490630 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:12:57.490687 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:12:57.490959 (+ 272us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:12:57.490960 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:12:57.490961 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:12:57.491114 (+ 153us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:12:57.491400 (+ 286us) server_negotiation.cc:1035] Waiting for connection context
0621 05:12:57.491639 (+ 239us) server_negotiation.cc:299] Negotiation successful
0621 05:12:57.491764 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260621 05:12:57.494812 16978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:12:57.484944 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.131.188:41779 (local address 127.0.0.1:57776)
0621 05:12:57.484994 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0621 05:12:57.485006 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0621 05:12:57.485041 (+ 35us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:12:57.485596 (+ 555us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:12:57.485597 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:12:57.485844 (+ 247us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:12:57.485881 (+ 37us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:12:57.485883 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:12:57.486991 (+ 1108us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:12:57.486992 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:12:57.487831 (+ 839us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:12:57.487834 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:12:57.488346 (+ 512us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:12:57.488347 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:12:57.488412 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:12:57.488881 (+ 469us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 05:12:57.488885 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:12:57.490486 (+ 1601us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 05:12:57.493713 (+ 3227us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:12:57.493714 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:12:57.493714 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:12:57.493917 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:12:57.494146 (+ 229us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:12:57.494147 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:12:57.494148 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:12:57.494211 (+ 63us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:12:57.494476 (+ 265us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 05:12:57.494477 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 05:12:57.494536 (+ 59us) client_negotiation.cc:769] Sending connection context
0621 05:12:57.494614 (+ 78us) client_negotiation.cc:240] Negotiation successful
0621 05:12:57.494720 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":28}
I20260621 05:12:57.495137 17129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:12:57.485047 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57776 (local address 127.3.131.188:41779)
0621 05:12:57.485474 (+ 427us) server_negotiation.cc:206] Beginning negotiation
0621 05:12:57.485476 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:12:57.485485 (+ 9us) server_negotiation.cc:407] Connection header received
0621 05:12:57.485507 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:12:57.485507 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:12:57.485518 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:12:57.485571 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:12:57.485976 (+ 405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:12:57.486882 (+ 906us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:12:57.487934 (+ 1052us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:12:57.488230 (+ 296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:12:57.488512 (+ 282us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:12:57.490872 (+ 2360us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:12:57.490873 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:12:57.490874 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:12:57.490882 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:12:57.493631 (+ 2749us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:12:57.493994 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:12:57.493996 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:12:57.493996 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:12:57.494060 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:12:57.494277 (+ 217us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:12:57.494278 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:12:57.494278 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:12:57.494391 (+ 113us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:12:57.494452 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0621 05:12:57.494970 (+ 518us) server_negotiation.cc:299] Negotiation successful
0621 05:12:57.495052 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260621 05:12:57.497382 16976 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7b5dd37dd079413f8c0ab35ffb1451e9: Bootstrap starting.
Jun 21 05:12:57 dist-test-slave-vrq1 krb5kdc[16895](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782018777, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.131.188@KRBTEST.COM for kudu/127.3.131.188@KRBTEST.COM
I20260621 05:12:57.498242 16976 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7b5dd37dd079413f8c0ab35ffb1451e9: Neither blocks nor log segments found. Creating new log.
I20260621 05:12:57.498589 16976 log.cc:824] T 00000000000000000000000000000000 P 7b5dd37dd079413f8c0ab35ffb1451e9: Log is configured to *not* fsync() on all Append() calls
Jun 21 05:12:57 dist-test-slave-vrq1 krb5kdc[16895](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782018776, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.131.189@KRBTEST.COM
I20260621 05:12:57.499516 16976 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7b5dd37dd079413f8c0ab35ffb1451e9: No bootstrap required, opened a new log
I20260621 05:12:57.501658 16976 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7b5dd37dd079413f8c0ab35ffb1451e9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b5dd37dd079413f8c0ab35ffb1451e9" member_type: VOTER last_known_addr { host: "127.3.131.190" port: 41507 } } peers { permanent_uuid: "7c7ba507fa614e6396d258655e2f59d8" member_type: VOTER last_known_addr { host: "127.3.131.189" port: 37745 } } peers { permanent_uuid: "8d1f2a82c8194921b40e47d32864c72d" member_type: VOTER last_known_addr { host: "127.3.131.188" port: 41779 } }
I20260621 05:12:57.501835 16976 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7b5dd37dd079413f8c0ab35ffb1451e9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 05:12:57.501925 16976 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7b5dd37dd079413f8c0ab35ffb1451e9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7b5dd37dd079413f8c0ab35ffb1451e9, State: Initialized, Role: FOLLOWER
I20260621 05:12:57.502067 17130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:12:57.491627 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.131.188:41779 (local address 127.0.0.1:57792)
0621 05:12:57.491690 (+ 63us) negotiation.cc:107] Waiting for socket to connect
0621 05:12:57.491702 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0621 05:12:57.491787 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:12:57.492259 (+ 472us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:12:57.492261 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:12:57.492658 (+ 397us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:12:57.492717 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:12:57.492720 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:12:57.494571 (+ 1851us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:12:57.494572 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:12:57.495471 (+ 899us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:12:57.495473 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:12:57.495986 (+ 513us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:12:57.495987 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:12:57.496041 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:12:57.496534 (+ 493us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 05:12:57.496539 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:12:57.498421 (+ 1882us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 05:12:57.500487 (+ 2066us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:12:57.500489 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:12:57.500489 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:12:57.501011 (+ 522us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:12:57.501256 (+ 245us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:12:57.501257 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:12:57.501257 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:12:57.501331 (+ 74us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:12:57.501746 (+ 415us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 05:12:57.501747 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 05:12:57.501789 (+ 42us) client_negotiation.cc:769] Sending connection context
0621 05:12:57.501852 (+ 63us) client_negotiation.cc:240] Negotiation successful
0621 05:12:57.501967 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":43}
I20260621 05:12:57.502090 16976 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7b5dd37dd079413f8c0ab35ffb1451e9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b5dd37dd079413f8c0ab35ffb1451e9" member_type: VOTER last_known_addr { host: "127.3.131.190" port: 41507 } } peers { permanent_uuid: "7c7ba507fa614e6396d258655e2f59d8" member_type: VOTER last_known_addr { host: "127.3.131.189" port: 37745 } } peers { permanent_uuid: "8d1f2a82c8194921b40e47d32864c72d" member_type: VOTER last_known_addr { host: "127.3.131.188" port: 41779 } }
I20260621 05:12:57.502403 17131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:12:57.491662 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57792 (local address 127.3.131.188:41779)
0621 05:12:57.491983 (+ 321us) server_negotiation.cc:206] Beginning negotiation
0621 05:12:57.491985 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:12:57.491993 (+ 8us) server_negotiation.cc:407] Connection header received
0621 05:12:57.492117 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:12:57.492118 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:12:57.492141 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:12:57.492336 (+ 195us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:12:57.492990 (+ 654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:12:57.494497 (+ 1507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:12:57.495562 (+ 1065us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:12:57.495896 (+ 334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:12:57.495988 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:12:57.498531 (+ 2543us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:12:57.498533 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:12:57.498533 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:12:57.498541 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:12:57.500390 (+ 1849us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:12:57.501105 (+ 715us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:12:57.501106 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:12:57.501107 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:12:57.501176 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:12:57.501410 (+ 234us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:12:57.501411 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:12:57.501411 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:12:57.501629 (+ 218us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:12:57.501715 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0621 05:12:57.502176 (+ 461us) server_negotiation.cc:299] Negotiation successful
0621 05:12:57.502263 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":297,"thread_start_us":110,"threads_started":1}
I20260621 05:12:57.502471 16976 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7b5dd37dd079413f8c0ab35ffb1451e9 [sys.catalog]: configured and running, proceeding with master startup.
I20260621 05:12:57.502900 17133 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7b5dd37dd079413f8c0ab35ffb1451e9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b5dd37dd079413f8c0ab35ffb1451e9" member_type: VOTER last_known_addr { host: "127.3.131.190" port: 41507 } } peers { permanent_uuid: "7c7ba507fa614e6396d258655e2f59d8" member_type: VOTER last_known_addr { host: "127.3.131.189" port: 37745 } } peers { permanent_uuid: "8d1f2a82c8194921b40e47d32864c72d" member_type: VOTER last_known_addr { host: "127.3.131.188" port: 41779 } } }
I20260621 05:12:57.503091 17133 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7b5dd37dd079413f8c0ab35ffb1451e9 [sys.catalog]: This master's current role is: FOLLOWER
I20260621 05:12:57.504168 17126 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8d1f2a82c8194921b40e47d32864c72d: Bootstrap starting.
W20260621 05:12:57.504441 17144 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 7b5dd37dd079413f8c0ab35ffb1451e9: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260621 05:12:57.504529 17144 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260621 05:12:57.504915 17126 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8d1f2a82c8194921b40e47d32864c72d: Neither blocks nor log segments found. Creating new log.
I20260621 05:12:57.505337 17126 log.cc:824] T 00000000000000000000000000000000 P 8d1f2a82c8194921b40e47d32864c72d: Log is configured to *not* fsync() on all Append() calls
I20260621 05:12:57.506062 17055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:12:57.493659 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36058 (local address 127.3.131.189:37745)
0621 05:12:57.494002 (+ 343us) server_negotiation.cc:206] Beginning negotiation
0621 05:12:57.494003 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0621 05:12:57.494010 (+ 7us) server_negotiation.cc:407] Connection header received
0621 05:12:57.494028 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:12:57.494029 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:12:57.494037 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:12:57.494113 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:12:57.494716 (+ 603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:12:57.495574 (+ 858us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:12:57.496462 (+ 888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:12:57.496663 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:12:57.496750 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:12:57.499867 (+ 3117us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:12:57.499869 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:12:57.499869 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:12:57.499879 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:12:57.501601 (+ 1722us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:12:57.502008 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:12:57.502010 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:12:57.502010 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:12:57.502072 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:12:57.502309 (+ 237us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:12:57.502310 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:12:57.502310 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:12:57.502441 (+ 131us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:12:57.502611 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0621 05:12:57.505857 (+ 3246us) server_negotiation.cc:299] Negotiation successful
0621 05:12:57.505965 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":320}
I20260621 05:12:57.506237 17126 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8d1f2a82c8194921b40e47d32864c72d: No bootstrap required, opened a new log
I20260621 05:12:57.508158 3598 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260621 05:12:57.506071 16908 master_runner.cc:186] Error getting master registration for 127.3.131.189:37745: OK, instance_id { permanent_uuid: "7c7ba507fa614e6396d258655e2f59d8" instance_seqno: 1782018777296923 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260621 05:12:57.509124 16908 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260621 05:12:57.509047 17126 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8d1f2a82c8194921b40e47d32864c72d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b5dd37dd079413f8c0ab35ffb1451e9" member_type: VOTER last_known_addr { host: "127.3.131.190" port: 41507 } } peers { permanent_uuid: "7c7ba507fa614e6396d258655e2f59d8" member_type: VOTER last_known_addr { host: "127.3.131.189" port: 37745 } } peers { permanent_uuid: "8d1f2a82c8194921b40e47d32864c72d" member_type: VOTER last_known_addr { host: "127.3.131.188" port: 41779 } }
I20260621 05:12:57.509220 17126 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8d1f2a82c8194921b40e47d32864c72d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 05:12:57.509259 17126 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8d1f2a82c8194921b40e47d32864c72d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8d1f2a82c8194921b40e47d32864c72d, State: Initialized, Role: FOLLOWER
I20260621 05:12:57.509444 17126 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8d1f2a82c8194921b40e47d32864c72d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b5dd37dd079413f8c0ab35ffb1451e9" member_type: VOTER last_known_addr { host: "127.3.131.190" port: 41507 } } peers { permanent_uuid: "7c7ba507fa614e6396d258655e2f59d8" member_type: VOTER last_known_addr { host: "127.3.131.189" port: 37745 } } peers { permanent_uuid: "8d1f2a82c8194921b40e47d32864c72d" member_type: VOTER last_known_addr { host: "127.3.131.188" port: 41779 } }
I20260621 05:12:57.509843 17126 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8d1f2a82c8194921b40e47d32864c72d [sys.catalog]: configured and running, proceeding with master startup.
I20260621 05:12:57.510187 17151 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8d1f2a82c8194921b40e47d32864c72d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b5dd37dd079413f8c0ab35ffb1451e9" member_type: VOTER last_known_addr { host: "127.3.131.190" port: 41507 } } peers { permanent_uuid: "7c7ba507fa614e6396d258655e2f59d8" member_type: VOTER last_known_addr { host: "127.3.131.189" port: 37745 } } peers { permanent_uuid: "8d1f2a82c8194921b40e47d32864c72d" member_type: VOTER last_known_addr { host: "127.3.131.188" port: 41779 } } }
I20260621 05:12:57.510275 17151 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8d1f2a82c8194921b40e47d32864c72d [sys.catalog]: This master's current role is: FOLLOWER
W20260621 05:12:57.512656 17165 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 8d1f2a82c8194921b40e47d32864c72d: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260621 05:12:57.512809 17165 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260621 05:12:57.513370 17058 master_runner.cc:186] Error getting master registration for 127.3.131.189:37745: OK, instance_id { permanent_uuid: "7c7ba507fa614e6396d258655e2f59d8" instance_seqno: 1782018777296923 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260621 05:12:57.515755 17058 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260621 05:12:57.520507 16979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:12:57.509879 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59516 (local address 127.3.131.190:41507)
0621 05:12:57.509927 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0621 05:12:57.509929 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:12:57.511691 (+ 1762us) server_negotiation.cc:407] Connection header received
0621 05:12:57.511750 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:12:57.511752 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:12:57.511764 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:12:57.511846 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:12:57.512615 (+ 769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:12:57.513377 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:12:57.514459 (+ 1082us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:12:57.514752 (+ 293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:12:57.514820 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:12:57.517280 (+ 2460us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:12:57.517282 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:12:57.517283 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:12:57.517292 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:12:57.519088 (+ 1796us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:12:57.519559 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:12:57.519561 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:12:57.519561 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:12:57.519628 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:12:57.519863 (+ 235us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:12:57.519864 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:12:57.519864 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:12:57.519995 (+ 131us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:12:57.520081 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0621 05:12:57.520322 (+ 241us) server_negotiation.cc:299] Negotiation successful
0621 05:12:57.520405 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260621 05:12:57.525419 17055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:12:57.510384 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36074 (local address 127.3.131.189:37745)
0621 05:12:57.510433 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0621 05:12:57.510435 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:12:57.511463 (+ 1028us) server_negotiation.cc:407] Connection header received
0621 05:12:57.511596 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:12:57.511597 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:12:57.511611 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:12:57.511670 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:12:57.515367 (+ 3697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:12:57.516105 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:12:57.518216 (+ 2111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:12:57.518527 (+ 311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:12:57.518575 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:12:57.522277 (+ 3702us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:12:57.522279 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:12:57.522279 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:12:57.522288 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:12:57.524080 (+ 1792us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:12:57.524562 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:12:57.524564 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:12:57.524564 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:12:57.524624 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:12:57.524833 (+ 209us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:12:57.524834 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:12:57.524834 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:12:57.524965 (+ 131us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:12:57.525049 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0621 05:12:57.525236 (+ 187us) server_negotiation.cc:299] Negotiation successful
0621 05:12:57.525320 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260621 05:12:57.528194 17131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:12:57.510223 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57802 (local address 127.3.131.188:41779)
0621 05:12:57.510272 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0621 05:12:57.510274 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:12:57.520568 (+ 10294us) server_negotiation.cc:407] Connection header received
0621 05:12:57.520598 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:12:57.520599 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:12:57.520624 (+ 25us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:12:57.520675 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:12:57.521221 (+ 546us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:12:57.521990 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:12:57.523221 (+ 1231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:12:57.523571 (+ 350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:12:57.523652 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:12:57.525141 (+ 1489us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:12:57.525143 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:12:57.525144 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:12:57.525151 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:12:57.526908 (+ 1757us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:12:57.527310 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:12:57.527311 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:12:57.527312 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:12:57.527376 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:12:57.527593 (+ 217us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:12:57.527594 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:12:57.527594 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:12:57.527756 (+ 162us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:12:57.527862 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0621 05:12:57.528006 (+ 144us) server_negotiation.cc:299] Negotiation successful
0621 05:12:57.528097 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260621 05:12:57.535954 17051 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.131.188" port: 41779 } attempt: 3
Jun 21 05:12:57 dist-test-slave-vrq1 krb5kdc[16895](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782018777, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.131.189@KRBTEST.COM for kudu/127.3.131.188@KRBTEST.COM
I20260621 05:12:57.545300 17131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:12:57.536413 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57814 (local address 127.3.131.188:41779)
0621 05:12:57.536485 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0621 05:12:57.536487 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:12:57.536498 (+ 11us) server_negotiation.cc:407] Connection header received
0621 05:12:57.536527 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:12:57.536528 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:12:57.536540 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:12:57.536603 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:12:57.536977 (+ 374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:12:57.538178 (+ 1201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:12:57.539253 (+ 1075us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:12:57.539584 (+ 331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:12:57.539661 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:12:57.542156 (+ 2495us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:12:57.542158 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:12:57.542159 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:12:57.542168 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:12:57.544044 (+ 1876us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:12:57.544466 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:12:57.544467 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:12:57.544467 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:12:57.544524 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:12:57.544716 (+ 192us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:12:57.544717 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:12:57.544717 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:12:57.544863 (+ 146us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:12:57.544930 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0621 05:12:57.545119 (+ 189us) server_negotiation.cc:299] Negotiation successful
0621 05:12:57.545202 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260621 05:12:57.545313 17053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:12:57.536308 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.131.188:41779 (local address 127.0.0.1:57814)
0621 05:12:57.536369 (+ 61us) negotiation.cc:107] Waiting for socket to connect
0621 05:12:57.536385 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0621 05:12:57.536455 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:12:57.536624 (+ 169us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:12:57.536625 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:12:57.536823 (+ 198us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:12:57.536881 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:12:57.536883 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:12:57.538273 (+ 1390us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:12:57.538274 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:12:57.539123 (+ 849us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:12:57.539127 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:12:57.539668 (+ 541us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:12:57.539670 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:12:57.539737 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:12:57.540297 (+ 560us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 05:12:57.540302 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:12:57.542048 (+ 1746us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 05:12:57.544155 (+ 2107us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:12:57.544156 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:12:57.544156 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:12:57.544381 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:12:57.544594 (+ 213us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:12:57.544594 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:12:57.544595 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:12:57.544649 (+ 54us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:12:57.544938 (+ 289us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 05:12:57.544939 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 05:12:57.544999 (+ 60us) client_negotiation.cc:769] Sending connection context
0621 05:12:57.545080 (+ 81us) client_negotiation.cc:240] Negotiation successful
0621 05:12:57.545192 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":28}
I20260621 05:12:57.547472 17051 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7c7ba507fa614e6396d258655e2f59d8: Bootstrap starting.
I20260621 05:12:57.548271 17051 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7c7ba507fa614e6396d258655e2f59d8: Neither blocks nor log segments found. Creating new log.
I20260621 05:12:57.548588 17051 log.cc:824] T 00000000000000000000000000000000 P 7c7ba507fa614e6396d258655e2f59d8: Log is configured to *not* fsync() on all Append() calls
I20260621 05:12:57.549614 17051 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7c7ba507fa614e6396d258655e2f59d8: No bootstrap required, opened a new log
I20260621 05:12:57.551831 17051 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7c7ba507fa614e6396d258655e2f59d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b5dd37dd079413f8c0ab35ffb1451e9" member_type: VOTER last_known_addr { host: "127.3.131.190" port: 41507 } } peers { permanent_uuid: "7c7ba507fa614e6396d258655e2f59d8" member_type: VOTER last_known_addr { host: "127.3.131.189" port: 37745 } } peers { permanent_uuid: "8d1f2a82c8194921b40e47d32864c72d" member_type: VOTER last_known_addr { host: "127.3.131.188" port: 41779 } }
I20260621 05:12:57.552000 17051 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7c7ba507fa614e6396d258655e2f59d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 05:12:57.552037 17051 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7c7ba507fa614e6396d258655e2f59d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7c7ba507fa614e6396d258655e2f59d8, State: Initialized, Role: FOLLOWER
I20260621 05:12:57.552211 17051 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7c7ba507fa614e6396d258655e2f59d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b5dd37dd079413f8c0ab35ffb1451e9" member_type: VOTER last_known_addr { host: "127.3.131.190" port: 41507 } } peers { permanent_uuid: "7c7ba507fa614e6396d258655e2f59d8" member_type: VOTER last_known_addr { host: "127.3.131.189" port: 37745 } } peers { permanent_uuid: "8d1f2a82c8194921b40e47d32864c72d" member_type: VOTER last_known_addr { host: "127.3.131.188" port: 41779 } }
I20260621 05:12:57.552595 17051 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7c7ba507fa614e6396d258655e2f59d8 [sys.catalog]: configured and running, proceeding with master startup.
I20260621 05:12:57.552552 17168 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7c7ba507fa614e6396d258655e2f59d8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b5dd37dd079413f8c0ab35ffb1451e9" member_type: VOTER last_known_addr { host: "127.3.131.190" port: 41507 } } peers { permanent_uuid: "7c7ba507fa614e6396d258655e2f59d8" member_type: VOTER last_known_addr { host: "127.3.131.189" port: 37745 } } peers { permanent_uuid: "8d1f2a82c8194921b40e47d32864c72d" member_type: VOTER last_known_addr { host: "127.3.131.188" port: 41779 } } }
I20260621 05:12:57.552760 17168 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7c7ba507fa614e6396d258655e2f59d8 [sys.catalog]: This master's current role is: FOLLOWER
W20260621 05:12:57.555042 17179 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 7c7ba507fa614e6396d258655e2f59d8: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260621 05:12:57.555217 17179 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260621 05:12:57.557001 16983 master_runner.cc:263] No leader master found from master 7c7ba507fa614e6396d258655e2f59d8
I20260621 05:12:57.557063 16983 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260621 05:12:57.563122 17151 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8d1f2a82c8194921b40e47d32864c72d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260621 05:12:57.563359 17151 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8d1f2a82c8194921b40e47d32864c72d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b5dd37dd079413f8c0ab35ffb1451e9" member_type: VOTER last_known_addr { host: "127.3.131.190" port: 41507 } } peers { permanent_uuid: "7c7ba507fa614e6396d258655e2f59d8" member_type: VOTER last_known_addr { host: "127.3.131.189" port: 37745 } } peers { permanent_uuid: "8d1f2a82c8194921b40e47d32864c72d" member_type: VOTER last_known_addr { host: "127.3.131.188" port: 41779 } }
I20260621 05:12:57.563642 17151 leader_election.cc:288] T 00000000000000000000000000000000 P 8d1f2a82c8194921b40e47d32864c72d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7b5dd37dd079413f8c0ab35ffb1451e9 (127.3.131.190:41507), 7c7ba507fa614e6396d258655e2f59d8 (127.3.131.189:37745)
I20260621 05:12:57.564293 17025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "8d1f2a82c8194921b40e47d32864c72d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7c7ba507fa614e6396d258655e2f59d8" is_pre_election: true
I20260621 05:12:57.564464 17025 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 7c7ba507fa614e6396d258655e2f59d8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8d1f2a82c8194921b40e47d32864c72d in term 0.
I20260621 05:12:57.564703 17074 leader_election.cc:302] T 00000000000000000000000000000000 P 8d1f2a82c8194921b40e47d32864c72d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7c7ba507fa614e6396d258655e2f59d8, 8d1f2a82c8194921b40e47d32864c72d; no voters:
I20260621 05:12:57.565196 17151 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8d1f2a82c8194921b40e47d32864c72d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260621 05:12:57.565439 17151 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8d1f2a82c8194921b40e47d32864c72d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260621 05:12:57.565508 17151 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8d1f2a82c8194921b40e47d32864c72d [term 0 FOLLOWER]: Advancing to term 1
I20260621 05:12:57.565441 16950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "8d1f2a82c8194921b40e47d32864c72d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b5dd37dd079413f8c0ab35ffb1451e9" is_pre_election: true
I20260621 05:12:57.565657 16950 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 7b5dd37dd079413f8c0ab35ffb1451e9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8d1f2a82c8194921b40e47d32864c72d in term 0.
I20260621 05:12:57.566954 17151 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8d1f2a82c8194921b40e47d32864c72d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b5dd37dd079413f8c0ab35ffb1451e9" member_type: VOTER last_known_addr { host: "127.3.131.190" port: 41507 } } peers { permanent_uuid: "7c7ba507fa614e6396d258655e2f59d8" member_type: VOTER last_known_addr { host: "127.3.131.189" port: 37745 } } peers { permanent_uuid: "8d1f2a82c8194921b40e47d32864c72d" member_type: VOTER last_known_addr { host: "127.3.131.188" port: 41779 } }
I20260621 05:12:57.567135 17151 leader_election.cc:288] T 00000000000000000000000000000000 P 8d1f2a82c8194921b40e47d32864c72d [CANDIDATE]: Term 1 election: Requested vote from peers 7b5dd37dd079413f8c0ab35ffb1451e9 (127.3.131.190:41507), 7c7ba507fa614e6396d258655e2f59d8 (127.3.131.189:37745)
I20260621 05:12:57.567344 16950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "8d1f2a82c8194921b40e47d32864c72d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b5dd37dd079413f8c0ab35ffb1451e9"
I20260621 05:12:57.567435 16950 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7b5dd37dd079413f8c0ab35ffb1451e9 [term 0 FOLLOWER]: Advancing to term 1
I20260621 05:12:57.567627 17025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "8d1f2a82c8194921b40e47d32864c72d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7c7ba507fa614e6396d258655e2f59d8"
I20260621 05:12:57.567729 17025 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7c7ba507fa614e6396d258655e2f59d8 [term 0 FOLLOWER]: Advancing to term 1
I20260621 05:12:57.568681 16950 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 7b5dd37dd079413f8c0ab35ffb1451e9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8d1f2a82c8194921b40e47d32864c72d in term 1.
I20260621 05:12:57.568681 17025 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 7c7ba507fa614e6396d258655e2f59d8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8d1f2a82c8194921b40e47d32864c72d in term 1.
I20260621 05:12:57.568908 17077 leader_election.cc:302] T 00000000000000000000000000000000 P 8d1f2a82c8194921b40e47d32864c72d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7b5dd37dd079413f8c0ab35ffb1451e9, 8d1f2a82c8194921b40e47d32864c72d; no voters:
I20260621 05:12:57.569023 17151 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8d1f2a82c8194921b40e47d32864c72d [term 1 FOLLOWER]: Leader election won for term 1
I20260621 05:12:57.569217 17151 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8d1f2a82c8194921b40e47d32864c72d [term 1 LEADER]: Becoming Leader. State: Replica: 8d1f2a82c8194921b40e47d32864c72d, State: Running, Role: LEADER
I20260621 05:12:57.569331 17151 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8d1f2a82c8194921b40e47d32864c72d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b5dd37dd079413f8c0ab35ffb1451e9" member_type: VOTER last_known_addr { host: "127.3.131.190" port: 41507 } } peers { permanent_uuid: "7c7ba507fa614e6396d258655e2f59d8" member_type: VOTER last_known_addr { host: "127.3.131.189" port: 37745 } } peers { permanent_uuid: "8d1f2a82c8194921b40e47d32864c72d" member_type: VOTER last_known_addr { host: "127.3.131.188" port: 41779 } }
I20260621 05:12:57.570029 17151 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8d1f2a82c8194921b40e47d32864c72d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8d1f2a82c8194921b40e47d32864c72d. Latest consensus state: current_term: 1 leader_uuid: "8d1f2a82c8194921b40e47d32864c72d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b5dd37dd079413f8c0ab35ffb1451e9" member_type: VOTER last_known_addr { host: "127.3.131.190" port: 41507 } } peers { permanent_uuid: "7c7ba507fa614e6396d258655e2f59d8" member_type: VOTER last_known_addr { host: "127.3.131.189" port: 37745 } } peers { permanent_uuid: "8d1f2a82c8194921b40e47d32864c72d" member_type: VOTER last_known_addr { host: "127.3.131.188" port: 41779 } } }
I20260621 05:12:57.570180 17151 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8d1f2a82c8194921b40e47d32864c72d [sys.catalog]: This master's current role is: LEADER
I20260621 05:12:57.571074 17183 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260621 05:12:57.571362 17183 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260621 05:12:57.573518 17025 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 7c7ba507fa614e6396d258655e2f59d8 [term 1 FOLLOWER]: Refusing update from remote peer 8d1f2a82c8194921b40e47d32864c72d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 05:12:57.574008 17181 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 8d1f2a82c8194921b40e47d32864c72d [LEADER]: Connected to new peer: Peer: permanent_uuid: "7c7ba507fa614e6396d258655e2f59d8" member_type: VOTER last_known_addr { host: "127.3.131.189" port: 37745 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 05:12:57.575639 17168 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7c7ba507fa614e6396d258655e2f59d8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8d1f2a82c8194921b40e47d32864c72d. Latest consensus state: current_term: 1 leader_uuid: "8d1f2a82c8194921b40e47d32864c72d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b5dd37dd079413f8c0ab35ffb1451e9" member_type: VOTER last_known_addr { host: "127.3.131.190" port: 41507 } } peers { permanent_uuid: "7c7ba507fa614e6396d258655e2f59d8" member_type: VOTER last_known_addr { host: "127.3.131.189" port: 37745 } } peers { permanent_uuid: "8d1f2a82c8194921b40e47d32864c72d" member_type: VOTER last_known_addr { host: "127.3.131.188" port: 41779 } } }
I20260621 05:12:57.575748 17168 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7c7ba507fa614e6396d258655e2f59d8 [sys.catalog]: This master's current role is: FOLLOWER
I20260621 05:12:57.576288 16950 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 7b5dd37dd079413f8c0ab35ffb1451e9 [term 1 FOLLOWER]: Refusing update from remote peer 8d1f2a82c8194921b40e47d32864c72d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 05:12:57.576640 17181 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 8d1f2a82c8194921b40e47d32864c72d [LEADER]: Connected to new peer: Peer: permanent_uuid: "7b5dd37dd079413f8c0ab35ffb1451e9" member_type: VOTER last_known_addr { host: "127.3.131.190" port: 41507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 05:12:57.578120 17133 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7b5dd37dd079413f8c0ab35ffb1451e9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8d1f2a82c8194921b40e47d32864c72d. Latest consensus state: current_term: 1 leader_uuid: "8d1f2a82c8194921b40e47d32864c72d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b5dd37dd079413f8c0ab35ffb1451e9" member_type: VOTER last_known_addr { host: "127.3.131.190" port: 41507 } } peers { permanent_uuid: "7c7ba507fa614e6396d258655e2f59d8" member_type: VOTER last_known_addr { host: "127.3.131.189" port: 37745 } } peers { permanent_uuid: "8d1f2a82c8194921b40e47d32864c72d" member_type: VOTER last_known_addr { host: "127.3.131.188" port: 41779 } } }
I20260621 05:12:57.578225 17133 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7b5dd37dd079413f8c0ab35ffb1451e9 [sys.catalog]: This master's current role is: FOLLOWER
I20260621 05:12:57.579912 17181 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8d1f2a82c8194921b40e47d32864c72d [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "8d1f2a82c8194921b40e47d32864c72d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b5dd37dd079413f8c0ab35ffb1451e9" member_type: VOTER last_known_addr { host: "127.3.131.190" port: 41507 } } peers { permanent_uuid: "7c7ba507fa614e6396d258655e2f59d8" member_type: VOTER last_known_addr { host: "127.3.131.189" port: 37745 } } peers { permanent_uuid: "8d1f2a82c8194921b40e47d32864c72d" member_type: VOTER last_known_addr { host: "127.3.131.188" port: 41779 } } }
I20260621 05:12:57.580015 17181 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8d1f2a82c8194921b40e47d32864c72d [sys.catalog]: This master's current role is: LEADER
I20260621 05:12:57.580667 17168 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7c7ba507fa614e6396d258655e2f59d8 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "8d1f2a82c8194921b40e47d32864c72d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b5dd37dd079413f8c0ab35ffb1451e9" member_type: VOTER last_known_addr { host: "127.3.131.190" port: 41507 } } peers { permanent_uuid: "7c7ba507fa614e6396d258655e2f59d8" member_type: VOTER last_known_addr { host: "127.3.131.189" port: 37745 } } peers { permanent_uuid: "8d1f2a82c8194921b40e47d32864c72d" member_type: VOTER last_known_addr { host: "127.3.131.188" port: 41779 } } }
I20260621 05:12:57.580760 17168 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7c7ba507fa614e6396d258655e2f59d8 [sys.catalog]: This master's current role is: FOLLOWER
I20260621 05:12:57.581060 17184 mvcc.cc:205] Tried to move back new op lower bound from 7299148912937635840 to 7299148912924643328. Current Snapshot: MvccSnapshot[applied={T|T < 7299148912937635840}]
I20260621 05:12:57.581223 17185 mvcc.cc:205] Tried to move back new op lower bound from 7299148912937635840 to 7299148912924643328. Current Snapshot: MvccSnapshot[applied={T|T < 7299148912937635840}]
I20260621 05:12:57.582990 17151 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8d1f2a82c8194921b40e47d32864c72d [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "8d1f2a82c8194921b40e47d32864c72d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b5dd37dd079413f8c0ab35ffb1451e9" member_type: VOTER last_known_addr { host: "127.3.131.190" port: 41507 } } peers { permanent_uuid: "7c7ba507fa614e6396d258655e2f59d8" member_type: VOTER last_known_addr { host: "127.3.131.189" port: 37745 } } peers { permanent_uuid: "8d1f2a82c8194921b40e47d32864c72d" member_type: VOTER last_known_addr { host: "127.3.131.188" port: 41779 } } }
I20260621 05:12:57.583089 17151 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8d1f2a82c8194921b40e47d32864c72d [sys.catalog]: This master's current role is: LEADER
I20260621 05:12:57.583734 17183 catalog_manager.cc:1347] Generated new cluster ID: 176d4870150d4005ac890757d825f6fe
I20260621 05:12:57.583787 17183 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260621 05:12:57.583738 17133 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7b5dd37dd079413f8c0ab35ffb1451e9 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "8d1f2a82c8194921b40e47d32864c72d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b5dd37dd079413f8c0ab35ffb1451e9" member_type: VOTER last_known_addr { host: "127.3.131.190" port: 41507 } } peers { permanent_uuid: "7c7ba507fa614e6396d258655e2f59d8" member_type: VOTER last_known_addr { host: "127.3.131.189" port: 37745 } } peers { permanent_uuid: "8d1f2a82c8194921b40e47d32864c72d" member_type: VOTER last_known_addr { host: "127.3.131.188" port: 41779 } } }
I20260621 05:12:57.583829 17133 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7b5dd37dd079413f8c0ab35ffb1451e9 [sys.catalog]: This master's current role is: FOLLOWER
I20260621 05:12:57.584093 17187 mvcc.cc:205] Tried to move back new op lower bound from 7299148912937635840 to 7299148912924643328. Current Snapshot: MvccSnapshot[applied={T|T < 7299148912937635840}]
I20260621 05:12:57.611750 17183 catalog_manager.cc:1370] Generated new certificate authority record
I20260621 05:12:57.612661 17183 catalog_manager.cc:1504] Loading token signing keys...
I20260621 05:12:57.618862 17183 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8d1f2a82c8194921b40e47d32864c72d: Generated new TSK 0
I20260621 05:12:57.619168 17183 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260621 05:12:58.505020 17144 catalog_manager.cc:1259] Loaded cluster ID: 176d4870150d4005ac890757d825f6fe
I20260621 05:12:58.505112 17144 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 7b5dd37dd079413f8c0ab35ffb1451e9: loading cluster ID for follower catalog manager: success
I20260621 05:12:58.505932 17144 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 7b5dd37dd079413f8c0ab35ffb1451e9: acquiring CA information for follower catalog manager: success
I20260621 05:12:58.506217 17144 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 7b5dd37dd079413f8c0ab35ffb1451e9: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260621 05:12:58.555787 17179 catalog_manager.cc:1259] Loaded cluster ID: 176d4870150d4005ac890757d825f6fe
I20260621 05:12:58.555876 17179 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 7c7ba507fa614e6396d258655e2f59d8: loading cluster ID for follower catalog manager: success
I20260621 05:12:58.556747 17179 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 7c7ba507fa614e6396d258655e2f59d8: acquiring CA information for follower catalog manager: success
I20260621 05:12:58.557085 17179 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 7c7ba507fa614e6396d258655e2f59d8: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260621 05:12:58.673251 3598 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWZOx2_/build/release/bin/kudu with pid 16908
I20260621 05:12:58.679033 3598 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWZOx2_/build/release/bin/kudu with pid 16983
I20260621 05:12:58.684718 3598 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWZOx2_/build/release/bin/kudu with pid 17058
2026-06-21T05:12:58Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (2866 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1782018616648578-3598-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 21 05:12:58 dist-test-slave-vrq1 krb5kdc[17198](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 21 05:12:58 dist-test-slave-vrq1 krb5kdc[17198](info): set up 2 sockets
Jun 21 05:12:58 dist-test-slave-vrq1 krb5kdc[17198](info): commencing operation
krb5kdc: starting...
W20260621 05:12:59.755554 3598 mini_kdc.cc:121] Time spent starting KDC: real 1.050s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1782018616648578-3598-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1782018616648578-3598-0/krb5kdc/test-user.keytab.
Jun 21 05:12:59 dist-test-slave-vrq1 krb5kdc[17198](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782018779, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-21T05:12:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-21T05:12:59Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.131.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.131.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.131.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1782018616648578-3598-0/krb5kdc/kudu_127.3.131.190.keytab.
Entry for principal kudu/127.3.131.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1782018616648578-3598-0/krb5kdc/kudu_127.3.131.190.keytab.
I20260621 05:12:59.882383 3598 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWZOx2_/build/release/bin/kudu
/tmp/dist-test-taskWZOx2_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1782018616648578-3598-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1782018616648578-3598-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1782018616648578-3598-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1782018616648578-3598-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.131.190:39095
--webserver_interface=127.3.131.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1782018616648578-3598-0/krb5kdc/kudu_127.3.131.190.keytab
--principal=kudu/127.3.131.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.131.148:38951
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.131.190:39095
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1782018616648578-3598-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1782018616648578-3598-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1782018616648578-3598-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 05:12:59.969969 17211 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 05:12:59.970224 17211 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 05:12:59.970252 17211 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 05:12:59.971640 17211 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 05:12:59.971692 17211 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 05:12:59.971709 17211 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 05:12:59.971724 17211 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 05:12:59.971737 17211 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 05:12:59.973400 17211 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.131.148:38951
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1782018616648578-3598-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1782018616648578-3598-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.131.190:39095
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1782018616648578-3598-0/krb5kdc/kudu_127.3.131.190.keytab
--principal=kudu/127.3.131.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.131.190:39095
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1782018616648578-3598-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.131.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17211
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1782018616648578-3598-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 21 Jun 2026 04:16:16 UTC on ccycloud.quasar-xycaha.root.comops.site
build id 4457881
I20260621 05:12:59.973606 17211 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 05:12:59.973798 17211 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 05:12:59.979024 17216 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:12:59.978986 17219 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:12:59.978986 17217 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:12:59.979591 17211 server_base.cc:1034] running on GCE node
I20260621 05:12:59.979933 17211 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 05:12:59.980297 17211 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 05:12:59.981475 17211 hybrid_clock.cc:648] HybridClock initialized: now 1782018779981477 us; error 48 us; skew 500 ppm
Jun 21 05:12:59 dist-test-slave-vrq1 krb5kdc[17198](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782018779, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.131.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 05:12:59.984457 17211 init.cc:377] Logged in from keytab as kudu/127.3.131.190@KRBTEST.COM (short username kudu)
I20260621 05:12:59.984951 17211 webserver.cc:460] Webserver started at http://127.3.131.190:38153/ using document root <none> and password file <none>
I20260621 05:12:59.985212 17211 fs_manager.cc:359] Metadata directory not provided
I20260621 05:12:59.985285 17211 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 05:12:59.985438 17211 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 05:12:59.986604 17211 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1782018616648578-3598-0/minicluster-data/master-0/data/instance:
uuid: "f90217df108146dfa4db7c5386442566"
format_stamp: "Formatted at 2026-06-21 05:12:59 on dist-test-slave-vrq1"
server_key: "36b4c4e93595f244875767ca375ea6c5"
server_key_iv: "7dea440fc7bddd3f968b6f5a3f22e96c"
server_key_version: "encryptionkey@0"
I20260621 05:12:59.986991 17211 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1782018616648578-3598-0/minicluster-data/master-0/wal/instance:
uuid: "f90217df108146dfa4db7c5386442566"
format_stamp: "Formatted at 2026-06-21 05:12:59 on dist-test-slave-vrq1"
server_key: "36b4c4e93595f244875767ca375ea6c5"
server_key_iv: "7dea440fc7bddd3f968b6f5a3f22e96c"
server_key_version: "encryptionkey@0"
I20260621 05:12:59.988493 17211 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260621 05:12:59.989358 17226 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 05:12:59.989580 17211 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260621 05:12:59.989688 17211 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1782018616648578-3598-0/minicluster-data/master-0/data,/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1782018616648578-3598-0/minicluster-data/master-0/wal
uuid: "f90217df108146dfa4db7c5386442566"
format_stamp: "Formatted at 2026-06-21 05:12:59 on dist-test-slave-vrq1"
server_key: "36b4c4e93595f244875767ca375ea6c5"
server_key_iv: "7dea440fc7bddd3f968b6f5a3f22e96c"
server_key_version: "encryptionkey@0"
I20260621 05:12:59.989766 17211 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1782018616648578-3598-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1782018616648578-3598-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1782018616648578-3598-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 05:13:00.005237 17211 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 05:13:00.007985 17211 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 05:13:00.008173 17211 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 05:13:00.012370 17211 rpc_server.cc:307] RPC server started. Bound to: 127.3.131.190:39095
I20260621 05:13:00.012414 17278 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.131.190:39095 every 8 connection(s)
I20260621 05:13:00.012826 17211 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1782018616648578-3598-0/minicluster-data/master-0/data/info.pb
I20260621 05:13:00.013794 17279 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 05:13:00.016357 17279 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f90217df108146dfa4db7c5386442566: Bootstrap starting.
I20260621 05:13:00.016966 17279 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f90217df108146dfa4db7c5386442566: Neither blocks nor log segments found. Creating new log.
I20260621 05:13:00.017218 17279 log.cc:824] T 00000000000000000000000000000000 P f90217df108146dfa4db7c5386442566: Log is configured to *not* fsync() on all Append() calls
I20260621 05:13:00.017954 3598 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWZOx2_/build/release/bin/kudu as pid 17211
I20260621 05:13:00.018010 17279 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f90217df108146dfa4db7c5386442566: No bootstrap required, opened a new log
I20260621 05:13:00.018081 3598 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1782018616648578-3598-0/minicluster-data/master-0/wal/instance
I20260621 05:13:00.018277 3598 external_mini_cluster.cc:1428] Setting key 1c9eeec31fbfd86ead7d4de01d748cef
I20260621 05:13:00.020381 17279 raft_consensus.cc:348] T 00000000000000000000000000000000 P f90217df108146dfa4db7c5386442566 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f90217df108146dfa4db7c5386442566" member_type: VOTER last_known_addr { host: "127.3.131.190" port: 39095 } }
I20260621 05:13:00.020581 17279 raft_consensus.cc:374] T 00000000000000000000000000000000 P f90217df108146dfa4db7c5386442566 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 05:13:00.020619 17279 raft_consensus.cc:729] T 00000000000000000000000000000000 P f90217df108146dfa4db7c5386442566 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f90217df108146dfa4db7c5386442566, State: Initialized, Role: FOLLOWER
I20260621 05:13:00.020793 17279 consensus_queue.cc:260] T 00000000000000000000000000000000 P f90217df108146dfa4db7c5386442566 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f90217df108146dfa4db7c5386442566" member_type: VOTER last_known_addr { host: "127.3.131.190" port: 39095 } }
I20260621 05:13:00.020889 17279 raft_consensus.cc:388] T 00000000000000000000000000000000 P f90217df108146dfa4db7c5386442566 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260621 05:13:00.020922 17279 raft_consensus.cc:482] T 00000000000000000000000000000000 P f90217df108146dfa4db7c5386442566 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260621 05:13:00.020991 17279 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f90217df108146dfa4db7c5386442566 [term 0 FOLLOWER]: Advancing to term 1
I20260621 05:13:00.021858 17279 raft_consensus.cc:504] T 00000000000000000000000000000000 P f90217df108146dfa4db7c5386442566 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f90217df108146dfa4db7c5386442566" member_type: VOTER last_known_addr { host: "127.3.131.190" port: 39095 } }
I20260621 05:13:00.021981 17279 leader_election.cc:302] T 00000000000000000000000000000000 P f90217df108146dfa4db7c5386442566 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f90217df108146dfa4db7c5386442566; no voters:
I20260621 05:13:00.022162 17279 leader_election.cc:288] T 00000000000000000000000000000000 P f90217df108146dfa4db7c5386442566 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260621 05:13:00.022451 17284 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f90217df108146dfa4db7c5386442566 [term 1 FOLLOWER]: Leader election won for term 1
I20260621 05:13:00.022459 17279 sys_catalog.cc:564] T 00000000000000000000000000000000 P f90217df108146dfa4db7c5386442566 [sys.catalog]: configured and running, proceeding with master startup.
I20260621 05:13:00.023545 17284 raft_consensus.cc:686] T 00000000000000000000000000000000 P f90217df108146dfa4db7c5386442566 [term 1 LEADER]: Becoming Leader. State: Replica: f90217df108146dfa4db7c5386442566, State: Running, Role: LEADER
I20260621 05:13:00.023670 17284 consensus_queue.cc:237] T 00000000000000000000000000000000 P f90217df108146dfa4db7c5386442566 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f90217df108146dfa4db7c5386442566" member_type: VOTER last_known_addr { host: "127.3.131.190" port: 39095 } }
I20260621 05:13:00.024358 17285 sys_catalog.cc:455] T 00000000000000000000000000000000 P f90217df108146dfa4db7c5386442566 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f90217df108146dfa4db7c5386442566" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f90217df108146dfa4db7c5386442566" member_type: VOTER last_known_addr { host: "127.3.131.190" port: 39095 } } }
I20260621 05:13:00.024457 17285 sys_catalog.cc:458] T 00000000000000000000000000000000 P f90217df108146dfa4db7c5386442566 [sys.catalog]: This master's current role is: LEADER
I20260621 05:13:00.024361 17291 sys_catalog.cc:455] T 00000000000000000000000000000000 P f90217df108146dfa4db7c5386442566 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f90217df108146dfa4db7c5386442566. Latest consensus state: current_term: 1 leader_uuid: "f90217df108146dfa4db7c5386442566" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f90217df108146dfa4db7c5386442566" member_type: VOTER last_known_addr { host: "127.3.131.190" port: 39095 } } }
I20260621 05:13:00.025117 17291 sys_catalog.cc:458] T 00000000000000000000000000000000 P f90217df108146dfa4db7c5386442566 [sys.catalog]: This master's current role is: LEADER
W20260621 05:13:00.024895 17298 catalog_manager.cc:1558] T 00000000000000000000000000000000 P f90217df108146dfa4db7c5386442566: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260621 05:13:00.025197 17298 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260621 05:13:00.025309 17299 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260621 05:13:00.025491 17299 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260621 05:13:00.027441 17299 catalog_manager.cc:1347] Generated new cluster ID: 3c2929c58451465f948cd741c11dc7e8
I20260621 05:13:00.027505 17299 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Jun 21 05:13:00 dist-test-slave-vrq1 krb5kdc[17198](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782018779, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.131.190@KRBTEST.COM
I20260621 05:13:00.032310 17283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:13:00.018885 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55650 (local address 127.3.131.190:39095)
0621 05:13:00.019125 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0621 05:13:00.019137 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0621 05:13:00.019189 (+ 52us) server_negotiation.cc:407] Connection header received
0621 05:13:00.020319 (+ 1130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:13:00.020329 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:13:00.020401 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:13:00.020570 (+ 169us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:13:00.021394 (+ 824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:13:00.023000 (+ 1606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:13:00.024357 (+ 1357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:13:00.025220 (+ 863us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:13:00.025356 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:13:00.028584 (+ 3228us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:13:00.028589 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:13:00.028589 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:13:00.028606 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:13:00.030530 (+ 1924us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:13:00.030968 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:13:00.030971 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:13:00.030971 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:13:00.031086 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:13:00.031362 (+ 276us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:13:00.031363 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:13:00.031364 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:13:00.031640 (+ 276us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:13:00.031849 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0621 05:13:00.031935 (+ 86us) server_negotiation.cc:299] Negotiation successful
0621 05:13:00.032098 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":182,"thread_start_us":122,"threads_started":1}
I20260621 05:13:00.033314 3598 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260621 05:13:00.041630 17299 catalog_manager.cc:1370] Generated new certificate authority record
I20260621 05:13:00.042392 17299 catalog_manager.cc:1504] Loading token signing keys...
I20260621 05:13:00.046838 17299 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f90217df108146dfa4db7c5386442566: Generated new TSK 0
I20260621 05:13:00.047122 17299 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260621 05:13:00.087446 3598 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWZOx2_/build/release/bin/kudu with pid 17211
2026-06-21T05:13:00Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (1400 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782018616648578-3598-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 21 05:13:00 dist-test-slave-vrq1 krb5kdc[17306](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 21 05:13:00 dist-test-slave-vrq1 krb5kdc[17306](info): set up 2 sockets
Jun 21 05:13:00 dist-test-slave-vrq1 krb5kdc[17306](info): commencing operation
krb5kdc: starting...
W20260621 05:13:01.156175 3598 mini_kdc.cc:121] Time spent starting KDC: real 1.050s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782018616648578-3598-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782018616648578-3598-0/krb5kdc/test-user.keytab.
Jun 21 05:13:01 dist-test-slave-vrq1 krb5kdc[17306](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782018781, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-21T05:13:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-21T05:13:01Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.131.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.131.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.131.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782018616648578-3598-0/krb5kdc/kudu_127.3.131.190.keytab.
Entry for principal kudu/127.3.131.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782018616648578-3598-0/krb5kdc/kudu_127.3.131.190.keytab.
I20260621 05:13:01.300136 3598 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWZOx2_/build/release/bin/kudu
/tmp/dist-test-taskWZOx2_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782018616648578-3598-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782018616648578-3598-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782018616648578-3598-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782018616648578-3598-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.131.190:45579
--webserver_interface=127.3.131.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782018616648578-3598-0/krb5kdc/kudu_127.3.131.190.keytab
--principal=kudu/127.3.131.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.131.148:41803
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.131.190:45579
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782018616648578-3598-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782018616648578-3598-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782018616648578-3598-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 05:13:01.389655 17319 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 05:13:01.390193 17319 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 05:13:01.390426 17319 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 05:13:01.391840 17319 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 05:13:01.391896 17319 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 05:13:01.391912 17319 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 05:13:01.391925 17319 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 05:13:01.391938 17319 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 05:13:01.394162 17319 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.131.148:41803
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782018616648578-3598-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782018616648578-3598-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.131.190:45579
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782018616648578-3598-0/krb5kdc/kudu_127.3.131.190.keytab
--principal=kudu/127.3.131.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.131.190:45579
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782018616648578-3598-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.131.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17319
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782018616648578-3598-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 21 Jun 2026 04:16:16 UTC on ccycloud.quasar-xycaha.root.comops.site
build id 4457881
I20260621 05:13:01.394435 17319 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 05:13:01.394635 17319 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 05:13:01.400669 17324 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:13:01.400678 17327 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:13:01.401391 17319 server_base.cc:1034] running on GCE node
W20260621 05:13:01.401512 17325 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:13:01.401799 17319 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 05:13:01.402189 17319 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 05:13:01.403424 17319 hybrid_clock.cc:648] HybridClock initialized: now 1782018781403393 us; error 53 us; skew 500 ppm
Jun 21 05:13:01 dist-test-slave-vrq1 krb5kdc[17306](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782018781, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.131.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 05:13:01.406582 17319 init.cc:377] Logged in from keytab as kudu/127.3.131.190@KRBTEST.COM (short username kudu)
I20260621 05:13:01.407599 17319 webserver.cc:460] Webserver started at http://127.3.131.190:37647/ using document root <none> and password file <none>
I20260621 05:13:01.408033 17319 fs_manager.cc:359] Metadata directory not provided
I20260621 05:13:01.408114 17319 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 05:13:01.408277 17319 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 05:13:01.409423 17319 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782018616648578-3598-0/minicluster-data/master-0/data/instance:
uuid: "f472f132d3144c2890a7a5b23ff3daca"
format_stamp: "Formatted at 2026-06-21 05:13:01 on dist-test-slave-vrq1"
server_key: "f0397a2c76cb9696b49ac32a35ead456"
server_key_iv: "ce600c98bb104273486781bd45d2ef0d"
server_key_version: "encryptionkey@0"
I20260621 05:13:01.409813 17319 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782018616648578-3598-0/minicluster-data/master-0/wal/instance:
uuid: "f472f132d3144c2890a7a5b23ff3daca"
format_stamp: "Formatted at 2026-06-21 05:13:01 on dist-test-slave-vrq1"
server_key: "f0397a2c76cb9696b49ac32a35ead456"
server_key_iv: "ce600c98bb104273486781bd45d2ef0d"
server_key_version: "encryptionkey@0"
I20260621 05:13:01.411593 17319 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.003s sys 0.000s
I20260621 05:13:01.413754 17334 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 05:13:01.414187 17319 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.002s sys 0.000s
I20260621 05:13:01.414458 17319 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782018616648578-3598-0/minicluster-data/master-0/data,/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782018616648578-3598-0/minicluster-data/master-0/wal
uuid: "f472f132d3144c2890a7a5b23ff3daca"
format_stamp: "Formatted at 2026-06-21 05:13:01 on dist-test-slave-vrq1"
server_key: "f0397a2c76cb9696b49ac32a35ead456"
server_key_iv: "ce600c98bb104273486781bd45d2ef0d"
server_key_version: "encryptionkey@0"
I20260621 05:13:01.414565 17319 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782018616648578-3598-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782018616648578-3598-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782018616648578-3598-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 05:13:01.433568 17319 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 05:13:01.436297 17319 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 05:13:01.436494 17319 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 05:13:01.440956 17319 rpc_server.cc:307] RPC server started. Bound to: 127.3.131.190:45579
I20260621 05:13:01.440997 17386 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.131.190:45579 every 8 connection(s)
I20260621 05:13:01.441387 17319 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782018616648578-3598-0/minicluster-data/master-0/data/info.pb
I20260621 05:13:01.442523 17387 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 05:13:01.445021 17387 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f472f132d3144c2890a7a5b23ff3daca: Bootstrap starting.
I20260621 05:13:01.445668 17387 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f472f132d3144c2890a7a5b23ff3daca: Neither blocks nor log segments found. Creating new log.
I20260621 05:13:01.445984 17387 log.cc:824] T 00000000000000000000000000000000 P f472f132d3144c2890a7a5b23ff3daca: Log is configured to *not* fsync() on all Append() calls
I20260621 05:13:01.446900 17387 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f472f132d3144c2890a7a5b23ff3daca: No bootstrap required, opened a new log
I20260621 05:13:01.447376 3598 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWZOx2_/build/release/bin/kudu as pid 17319
I20260621 05:13:01.447470 3598 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782018616648578-3598-0/minicluster-data/master-0/wal/instance
I20260621 05:13:01.447667 3598 external_mini_cluster.cc:1428] Setting key da1350065ce1bcbc9eb0e9001fc0fe7c
I20260621 05:13:01.449962 17387 raft_consensus.cc:348] T 00000000000000000000000000000000 P f472f132d3144c2890a7a5b23ff3daca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f472f132d3144c2890a7a5b23ff3daca" member_type: VOTER last_known_addr { host: "127.3.131.190" port: 45579 } }
I20260621 05:13:01.450263 17387 raft_consensus.cc:374] T 00000000000000000000000000000000 P f472f132d3144c2890a7a5b23ff3daca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 05:13:01.450317 17387 raft_consensus.cc:729] T 00000000000000000000000000000000 P f472f132d3144c2890a7a5b23ff3daca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f472f132d3144c2890a7a5b23ff3daca, State: Initialized, Role: FOLLOWER
I20260621 05:13:01.450523 17387 consensus_queue.cc:260] T 00000000000000000000000000000000 P f472f132d3144c2890a7a5b23ff3daca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f472f132d3144c2890a7a5b23ff3daca" member_type: VOTER last_known_addr { host: "127.3.131.190" port: 45579 } }
I20260621 05:13:01.450618 17387 raft_consensus.cc:388] T 00000000000000000000000000000000 P f472f132d3144c2890a7a5b23ff3daca [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260621 05:13:01.450644 17387 raft_consensus.cc:482] T 00000000000000000000000000000000 P f472f132d3144c2890a7a5b23ff3daca [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260621 05:13:01.450712 17387 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f472f132d3144c2890a7a5b23ff3daca [term 0 FOLLOWER]: Advancing to term 1
I20260621 05:13:01.452106 17387 raft_consensus.cc:504] T 00000000000000000000000000000000 P f472f132d3144c2890a7a5b23ff3daca [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f472f132d3144c2890a7a5b23ff3daca" member_type: VOTER last_known_addr { host: "127.3.131.190" port: 45579 } }
I20260621 05:13:01.452234 17387 leader_election.cc:302] T 00000000000000000000000000000000 P f472f132d3144c2890a7a5b23ff3daca [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f472f132d3144c2890a7a5b23ff3daca; no voters:
I20260621 05:13:01.452404 17387 leader_election.cc:288] T 00000000000000000000000000000000 P f472f132d3144c2890a7a5b23ff3daca [CANDIDATE]: Term 1 election: Requested vote from peers
I20260621 05:13:01.452524 17392 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f472f132d3144c2890a7a5b23ff3daca [term 1 FOLLOWER]: Leader election won for term 1
I20260621 05:13:01.452711 17387 sys_catalog.cc:564] T 00000000000000000000000000000000 P f472f132d3144c2890a7a5b23ff3daca [sys.catalog]: configured and running, proceeding with master startup.
I20260621 05:13:01.452744 17392 raft_consensus.cc:686] T 00000000000000000000000000000000 P f472f132d3144c2890a7a5b23ff3daca [term 1 LEADER]: Becoming Leader. State: Replica: f472f132d3144c2890a7a5b23ff3daca, State: Running, Role: LEADER
I20260621 05:13:01.453310 17392 consensus_queue.cc:237] T 00000000000000000000000000000000 P f472f132d3144c2890a7a5b23ff3daca [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f472f132d3144c2890a7a5b23ff3daca" member_type: VOTER last_known_addr { host: "127.3.131.190" port: 45579 } }
I20260621 05:13:01.454401 17393 sys_catalog.cc:455] T 00000000000000000000000000000000 P f472f132d3144c2890a7a5b23ff3daca [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f472f132d3144c2890a7a5b23ff3daca" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f472f132d3144c2890a7a5b23ff3daca" member_type: VOTER last_known_addr { host: "127.3.131.190" port: 45579 } } }
I20260621 05:13:01.454706 17393 sys_catalog.cc:458] T 00000000000000000000000000000000 P f472f132d3144c2890a7a5b23ff3daca [sys.catalog]: This master's current role is: LEADER
I20260621 05:13:01.455292 17394 sys_catalog.cc:455] T 00000000000000000000000000000000 P f472f132d3144c2890a7a5b23ff3daca [sys.catalog]: SysCatalogTable state changed. Reason: New leader f472f132d3144c2890a7a5b23ff3daca. Latest consensus state: current_term: 1 leader_uuid: "f472f132d3144c2890a7a5b23ff3daca" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f472f132d3144c2890a7a5b23ff3daca" member_type: VOTER last_known_addr { host: "127.3.131.190" port: 45579 } } }
I20260621 05:13:01.455513 17394 sys_catalog.cc:458] T 00000000000000000000000000000000 P f472f132d3144c2890a7a5b23ff3daca [sys.catalog]: This master's current role is: LEADER
W20260621 05:13:01.455868 17408 catalog_manager.cc:1558] T 00000000000000000000000000000000 P f472f132d3144c2890a7a5b23ff3daca: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260621 05:13:01.455928 17408 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260621 05:13:01.455977 17402 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260621 05:13:01.456226 17402 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Jun 21 05:13:01 dist-test-slave-vrq1 krb5kdc[17306](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782018781, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.131.190@KRBTEST.COM
I20260621 05:13:01.458302 17402 catalog_manager.cc:1347] Generated new cluster ID: 01c16c36d7014e3d8bdcd707caf5e679
I20260621 05:13:01.458436 17402 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260621 05:13:01.463912 17391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:13:01.448229 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39752 (local address 127.3.131.190:45579)
0621 05:13:01.448556 (+ 327us) server_negotiation.cc:206] Beginning negotiation
0621 05:13:01.448567 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0621 05:13:01.448616 (+ 49us) server_negotiation.cc:407] Connection header received
0621 05:13:01.449904 (+ 1288us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:13:01.449914 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:13:01.449990 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:13:01.450260 (+ 270us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:13:01.451082 (+ 822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:13:01.452611 (+ 1529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:13:01.454388 (+ 1777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:13:01.455309 (+ 921us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:13:01.455863 (+ 554us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:13:01.459041 (+ 3178us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:13:01.459046 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:13:01.459047 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:13:01.459059 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:13:01.461595 (+ 2536us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:13:01.462311 (+ 716us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:13:01.462315 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:13:01.462315 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:13:01.462470 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:13:01.462746 (+ 276us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:13:01.462747 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:13:01.462748 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:13:01.463075 (+ 327us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:13:01.463271 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0621 05:13:01.463409 (+ 138us) server_negotiation.cc:299] Negotiation successful
0621 05:13:01.463563 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":212,"thread_start_us":151,"threads_started":1}
I20260621 05:13:01.468724 17402 catalog_manager.cc:1370] Generated new certificate authority record
I20260621 05:13:01.469615 17402 catalog_manager.cc:1504] Loading token signing keys...
I20260621 05:13:01.480525 17402 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f472f132d3144c2890a7a5b23ff3daca: Generated new TSK 0
I20260621 05:13:01.481031 17402 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.3.131.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.131.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.131.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782018616648578-3598-0/krb5kdc/kudu_127.3.131.129.keytab.
Entry for principal kudu/127.3.131.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782018616648578-3598-0/krb5kdc/kudu_127.3.131.129.keytab.
I20260621 05:13:01.540290 3598 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWZOx2_/build/release/bin/kudu
/tmp/dist-test-taskWZOx2_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782018616648578-3598-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782018616648578-3598-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782018616648578-3598-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782018616648578-3598-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.131.129:0
--local_ip_for_outbound_sockets=127.3.131.129
--webserver_interface=127.3.131.129
--webserver_port=0
--tserver_master_addrs=127.3.131.190:45579
--keytab_file=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782018616648578-3598-0/krb5kdc/kudu_127.3.131.129.keytab
--principal=kudu/127.3.131.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.131.148:41803
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782018616648578-3598-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782018616648578-3598-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782018616648578-3598-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 05:13:01.642057 17413 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 05:13:01.642355 17413 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 05:13:01.642403 17413 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 05:13:01.643941 17413 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 05:13:01.644032 17413 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 05:13:01.644084 17413 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.131.129
I20260621 05:13:01.645867 17413 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.131.148:41803
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782018616648578-3598-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782018616648578-3598-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782018616648578-3598-0/krb5kdc/kudu_127.3.131.129.keytab
--principal=kudu/127.3.131.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.131.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782018616648578-3598-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.131.129
--webserver_port=0
--tserver_master_addrs=127.3.131.190:45579
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17413
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.131.129
--log_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782018616648578-3598-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 21 Jun 2026 04:16:16 UTC on ccycloud.quasar-xycaha.root.comops.site
build id 4457881
I20260621 05:13:01.646186 17413 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 05:13:01.646445 17413 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 05:13:01.652504 17418 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:13:01.652411 17419 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:13:01.652536 17421 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:13:01.652722 17413 server_base.cc:1034] running on GCE node
I20260621 05:13:01.652947 17413 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 05:13:01.653183 17413 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 05:13:01.654433 17413 hybrid_clock.cc:648] HybridClock initialized: now 1782018781654404 us; error 58 us; skew 500 ppm
Jun 21 05:13:01 dist-test-slave-vrq1 krb5kdc[17306](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782018781, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.131.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 05:13:01.657361 17413 init.cc:377] Logged in from keytab as kudu/127.3.131.129@KRBTEST.COM (short username kudu)
I20260621 05:13:01.657893 17413 webserver.cc:460] Webserver started at http://127.3.131.129:45015/ using document root <none> and password file <none>
I20260621 05:13:01.658164 17413 fs_manager.cc:359] Metadata directory not provided
I20260621 05:13:01.658306 17413 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 05:13:01.658563 17413 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 05:13:01.659822 17413 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782018616648578-3598-0/minicluster-data/ts-0/data/instance:
uuid: "828bb2f146144ed2826a65906ec39af5"
format_stamp: "Formatted at 2026-06-21 05:13:01 on dist-test-slave-vrq1"
server_key: "ef1e675f068b0c5dcdf64fe5a4bcbe36"
server_key_iv: "86e5551649e045a042c1f92215298a30"
server_key_version: "encryptionkey@0"
I20260621 05:13:01.660202 17413 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782018616648578-3598-0/minicluster-data/ts-0/wal/instance:
uuid: "828bb2f146144ed2826a65906ec39af5"
format_stamp: "Formatted at 2026-06-21 05:13:01 on dist-test-slave-vrq1"
server_key: "ef1e675f068b0c5dcdf64fe5a4bcbe36"
server_key_iv: "86e5551649e045a042c1f92215298a30"
server_key_version: "encryptionkey@0"
I20260621 05:13:01.661691 17413 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260621 05:13:01.662614 17428 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 05:13:01.662889 17413 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260621 05:13:01.663010 17413 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782018616648578-3598-0/minicluster-data/ts-0/data,/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782018616648578-3598-0/minicluster-data/ts-0/wal
uuid: "828bb2f146144ed2826a65906ec39af5"
format_stamp: "Formatted at 2026-06-21 05:13:01 on dist-test-slave-vrq1"
server_key: "ef1e675f068b0c5dcdf64fe5a4bcbe36"
server_key_iv: "86e5551649e045a042c1f92215298a30"
server_key_version: "encryptionkey@0"
I20260621 05:13:01.663096 17413 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782018616648578-3598-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782018616648578-3598-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782018616648578-3598-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 05:13:01.674050 17413 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 05:13:01.676370 17413 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 05:13:01.676570 17413 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 05:13:01.676873 17413 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 05:13:01.677204 17413 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 05:13:01.677258 17413 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 05:13:01.677292 17413 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 05:13:01.677320 17413 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 05:13:01.684068 17413 rpc_server.cc:307] RPC server started. Bound to: 127.3.131.129:45257
I20260621 05:13:01.684108 17541 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.131.129:45257 every 8 connection(s)
I20260621 05:13:01.684504 17413 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782018616648578-3598-0/minicluster-data/ts-0/data/info.pb
I20260621 05:13:01.687489 3598 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWZOx2_/build/release/bin/kudu as pid 17413
I20260621 05:13:01.687608 3598 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782018616648578-3598-0/minicluster-data/ts-0/wal/instance
I20260621 05:13:01.687816 3598 external_mini_cluster.cc:1428] Setting key c5344d752ca12677e7dc65cf8e96941c
Jun 21 05:13:01 dist-test-slave-vrq1 krb5kdc[17306](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782018781, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.131.129@KRBTEST.COM for kudu/127.3.131.190@KRBTEST.COM
I20260621 05:13:01.695434 17391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:13:01.685465 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.131.129:37683 (local address 127.3.131.190:45579)
0621 05:13:01.685520 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0621 05:13:01.685522 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:13:01.685761 (+ 239us) server_negotiation.cc:407] Connection header received
0621 05:13:01.686275 (+ 514us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:13:01.686277 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:13:01.686290 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:13:01.686354 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:13:01.687446 (+ 1092us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:13:01.688248 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:13:01.689407 (+ 1159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:13:01.689643 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:13:01.689723 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:13:01.692317 (+ 2594us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:13:01.692319 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:13:01.692319 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:13:01.692331 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:13:01.693864 (+ 1533us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:13:01.694340 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:13:01.694342 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:13:01.694342 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:13:01.694405 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:13:01.694680 (+ 275us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:13:01.694681 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:13:01.694682 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:13:01.694818 (+ 136us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:13:01.694899 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0621 05:13:01.695223 (+ 324us) server_negotiation.cc:299] Negotiation successful
0621 05:13:01.695322 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260621 05:13:01.695638 17544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:13:01.685345 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.131.190:45579 (local address 127.3.131.129:37683)
0621 05:13:01.685596 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0621 05:13:01.685642 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0621 05:13:01.686163 (+ 521us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:13:01.686447 (+ 284us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:13:01.686459 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:13:01.687049 (+ 590us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:13:01.687291 (+ 242us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:13:01.687296 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:13:01.688352 (+ 1056us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:13:01.688353 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:13:01.689295 (+ 942us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:13:01.689299 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:13:01.689732 (+ 433us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:13:01.689733 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:13:01.689887 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:13:01.690482 (+ 595us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 05:13:01.690491 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:13:01.692188 (+ 1697us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 05:13:01.693976 (+ 1788us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:13:01.693986 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:13:01.693988 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:13:01.694238 (+ 250us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:13:01.694477 (+ 239us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:13:01.694478 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:13:01.694479 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:13:01.694609 (+ 130us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:13:01.694888 (+ 279us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 05:13:01.694896 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 05:13:01.695053 (+ 157us) client_negotiation.cc:769] Sending connection context
0621 05:13:01.695169 (+ 116us) client_negotiation.cc:240] Negotiation successful
0621 05:13:01.695370 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":210,"thread_start_us":178,"threads_started":1}
I20260621 05:13:01.696153 17542 heartbeater.cc:347] Connected to a master server at 127.3.131.190:45579
I20260621 05:13:01.696305 17542 heartbeater.cc:464] Registering TS with master...
I20260621 05:13:01.696514 17542 heartbeater.cc:511] Master 127.3.131.190:45579 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.3.131.130@KRBTEST.COM; defaulting to no policy
I20260621 05:13:01.697221 17351 ts_manager.cc:194] Registered new tserver with Master: 828bb2f146144ed2826a65906ec39af5 (127.3.131.129:45257)
I20260621 05:13:01.697865 17351 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.131.129@KRBTEST.COM'} at 127.3.131.129:37683
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.131.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.131.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782018616648578-3598-0/krb5kdc/kudu_127.3.131.130.keytab.
Entry for principal kudu/127.3.131.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782018616648578-3598-0/krb5kdc/kudu_127.3.131.130.keytab.
I20260621 05:13:01.719842 3598 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWZOx2_/build/release/bin/kudu
/tmp/dist-test-taskWZOx2_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782018616648578-3598-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782018616648578-3598-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782018616648578-3598-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782018616648578-3598-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.131.130:0
--local_ip_for_outbound_sockets=127.3.131.130
--webserver_interface=127.3.131.130
--webserver_port=0
--tserver_master_addrs=127.3.131.190:45579
--keytab_file=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782018616648578-3598-0/krb5kdc/kudu_127.3.131.130.keytab
--principal=kudu/127.3.131.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.131.148:41803
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782018616648578-3598-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782018616648578-3598-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782018616648578-3598-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 05:13:01.812095 17547 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 05:13:01.812341 17547 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 05:13:01.812379 17547 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 05:13:01.813638 17547 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 05:13:01.813686 17547 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 05:13:01.813717 17547 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.131.130
I20260621 05:13:01.815454 17547 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.131.148:41803
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782018616648578-3598-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782018616648578-3598-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782018616648578-3598-0/krb5kdc/kudu_127.3.131.130.keytab
--principal=kudu/127.3.131.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.131.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782018616648578-3598-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.131.130
--webserver_port=0
--tserver_master_addrs=127.3.131.190:45579
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17547
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.131.130
--log_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782018616648578-3598-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 21 Jun 2026 04:16:16 UTC on ccycloud.quasar-xycaha.root.comops.site
build id 4457881
I20260621 05:13:01.815690 17547 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 05:13:01.815888 17547 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 05:13:01.821986 17553 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:13:01.821986 17552 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:13:01.822381 17555 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:13:01.822497 17547 server_base.cc:1034] running on GCE node
I20260621 05:13:01.822741 17547 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 05:13:01.823539 17547 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 05:13:01.824826 17547 hybrid_clock.cc:648] HybridClock initialized: now 1782018781824796 us; error 40 us; skew 500 ppm
Jun 21 05:13:01 dist-test-slave-vrq1 krb5kdc[17306](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782018781, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.131.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 05:13:01.828312 17547 init.cc:377] Logged in from keytab as kudu/127.3.131.130@KRBTEST.COM (short username kudu)
I20260621 05:13:01.828855 17547 webserver.cc:460] Webserver started at http://127.3.131.130:46151/ using document root <none> and password file <none>
I20260621 05:13:01.829150 17547 fs_manager.cc:359] Metadata directory not provided
I20260621 05:13:01.829207 17547 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 05:13:01.829361 17547 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 05:13:01.831261 17547 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782018616648578-3598-0/minicluster-data/ts-1/data/instance:
uuid: "090c4546eb4d4b02b3daa5bb23ec3444"
format_stamp: "Formatted at 2026-06-21 05:13:01 on dist-test-slave-vrq1"
server_key: "87ccbc83f2293f21959f5ce191249c7d"
server_key_iv: "766838c1befa0d8cc7fb0ec6b5f15826"
server_key_version: "encryptionkey@0"
I20260621 05:13:01.831686 17547 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782018616648578-3598-0/minicluster-data/ts-1/wal/instance:
uuid: "090c4546eb4d4b02b3daa5bb23ec3444"
format_stamp: "Formatted at 2026-06-21 05:13:01 on dist-test-slave-vrq1"
server_key: "87ccbc83f2293f21959f5ce191249c7d"
server_key_iv: "766838c1befa0d8cc7fb0ec6b5f15826"
server_key_version: "encryptionkey@0"
I20260621 05:13:01.833290 17547 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260621 05:13:01.834348 17562 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 05:13:01.834684 17547 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260621 05:13:01.834772 17547 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782018616648578-3598-0/minicluster-data/ts-1/data,/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782018616648578-3598-0/minicluster-data/ts-1/wal
uuid: "090c4546eb4d4b02b3daa5bb23ec3444"
format_stamp: "Formatted at 2026-06-21 05:13:01 on dist-test-slave-vrq1"
server_key: "87ccbc83f2293f21959f5ce191249c7d"
server_key_iv: "766838c1befa0d8cc7fb0ec6b5f15826"
server_key_version: "encryptionkey@0"
I20260621 05:13:01.834879 17547 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782018616648578-3598-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782018616648578-3598-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782018616648578-3598-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 05:13:01.870045 17547 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 05:13:01.873586 17547 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 05:13:01.873950 17547 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 05:13:01.874351 17547 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 05:13:01.874742 17547 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 05:13:01.874801 17547 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 05:13:01.874877 17547 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 05:13:01.874912 17547 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 05:13:01.882148 17547 rpc_server.cc:307] RPC server started. Bound to: 127.3.131.130:42965
I20260621 05:13:01.882179 17675 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.131.130:42965 every 8 connection(s)
I20260621 05:13:01.882560 17547 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782018616648578-3598-0/minicluster-data/ts-1/data/info.pb
I20260621 05:13:01.887054 3598 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWZOx2_/build/release/bin/kudu as pid 17547
I20260621 05:13:01.887202 3598 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782018616648578-3598-0/minicluster-data/ts-1/wal/instance
I20260621 05:13:01.887409 3598 external_mini_cluster.cc:1428] Setting key ade696a9d803150bbfb576cbbb0eb657
Jun 21 05:13:01 dist-test-slave-vrq1 krb5kdc[17306](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782018781, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.131.130@KRBTEST.COM for kudu/127.3.131.190@KRBTEST.COM
I20260621 05:13:01.897423 17391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:13:01.883831 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.131.130:39549 (local address 127.3.131.190:45579)
0621 05:13:01.883900 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0621 05:13:01.883908 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0621 05:13:01.884120 (+ 212us) server_negotiation.cc:407] Connection header received
0621 05:13:01.884689 (+ 569us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:13:01.884690 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:13:01.884701 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:13:01.884785 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:13:01.885982 (+ 1197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:13:01.886652 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:13:01.888775 (+ 2123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:13:01.889144 (+ 369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:13:01.889234 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:13:01.892872 (+ 3638us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:13:01.892874 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:13:01.892874 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:13:01.892885 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:13:01.894842 (+ 1957us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:13:01.895578 (+ 736us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:13:01.895580 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:13:01.895580 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:13:01.895660 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:13:01.896118 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:13:01.896120 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:13:01.896120 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:13:01.896432 (+ 312us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:13:01.896505 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0621 05:13:01.897081 (+ 576us) server_negotiation.cc:299] Negotiation successful
0621 05:13:01.897352 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260621 05:13:01.897513 17678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:13:01.883721 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.131.190:45579 (local address 127.3.131.130:39549)
0621 05:13:01.883941 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0621 05:13:01.883991 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0621 05:13:01.884552 (+ 561us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:13:01.884925 (+ 373us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:13:01.884929 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:13:01.885553 (+ 624us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:13:01.885791 (+ 238us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:13:01.885797 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:13:01.886841 (+ 1044us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:13:01.886844 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:13:01.888652 (+ 1808us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:13:01.888656 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:13:01.889258 (+ 602us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:13:01.889259 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:13:01.889550 (+ 291us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:13:01.890208 (+ 658us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 05:13:01.890220 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:13:01.892691 (+ 2471us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 05:13:01.894964 (+ 2273us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:13:01.894978 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:13:01.894982 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:13:01.895469 (+ 487us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:13:01.895775 (+ 306us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:13:01.895776 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:13:01.895777 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:13:01.895963 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:13:01.896575 (+ 612us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 05:13:01.896581 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 05:13:01.896826 (+ 245us) client_negotiation.cc:769] Sending connection context
0621 05:13:01.896966 (+ 140us) client_negotiation.cc:240] Negotiation successful
0621 05:13:01.897173 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":173,"thread_start_us":137,"threads_started":1}
I20260621 05:13:01.898181 17676 heartbeater.cc:347] Connected to a master server at 127.3.131.190:45579
WARNING: no policy specified for kudu/127.3.131.131@KRBTEST.COM; defaulting to no policy
I20260621 05:13:01.898409 17676 heartbeater.cc:464] Registering TS with master...
I20260621 05:13:01.898627 17676 heartbeater.cc:511] Master 127.3.131.190:45579 requested a full tablet report, sending...
I20260621 05:13:01.899421 17351 ts_manager.cc:194] Registered new tserver with Master: 090c4546eb4d4b02b3daa5bb23ec3444 (127.3.131.130:42965)
I20260621 05:13:01.899861 17351 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.131.130@KRBTEST.COM'} at 127.3.131.130:39549
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.131.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.131.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782018616648578-3598-0/krb5kdc/kudu_127.3.131.131.keytab.
Entry for principal kudu/127.3.131.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782018616648578-3598-0/krb5kdc/kudu_127.3.131.131.keytab.
I20260621 05:13:01.928150 3598 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWZOx2_/build/release/bin/kudu
/tmp/dist-test-taskWZOx2_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782018616648578-3598-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782018616648578-3598-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782018616648578-3598-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782018616648578-3598-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.131.131:0
--local_ip_for_outbound_sockets=127.3.131.131
--webserver_interface=127.3.131.131
--webserver_port=0
--tserver_master_addrs=127.3.131.190:45579
--keytab_file=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782018616648578-3598-0/krb5kdc/kudu_127.3.131.131.keytab
--principal=kudu/127.3.131.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.131.148:41803
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782018616648578-3598-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782018616648578-3598-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782018616648578-3598-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 05:13:02.025077 17681 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 05:13:02.025369 17681 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 05:13:02.025413 17681 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 05:13:02.026801 17681 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 05:13:02.026877 17681 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 05:13:02.026926 17681 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.131.131
I20260621 05:13:02.028684 17681 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.131.148:41803
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782018616648578-3598-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782018616648578-3598-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782018616648578-3598-0/krb5kdc/kudu_127.3.131.131.keytab
--principal=kudu/127.3.131.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.131.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782018616648578-3598-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.131.131
--webserver_port=0
--tserver_master_addrs=127.3.131.190:45579
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17681
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.131.131
--log_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782018616648578-3598-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 21 Jun 2026 04:16:16 UTC on ccycloud.quasar-xycaha.root.comops.site
build id 4457881
I20260621 05:13:02.029008 17681 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 05:13:02.029402 17681 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 05:13:02.034965 17687 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:13:02.034891 17689 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:13:02.034904 17686 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:13:02.035367 17681 server_base.cc:1034] running on GCE node
I20260621 05:13:02.035630 17681 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 05:13:02.035912 17681 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 05:13:02.037184 17681 hybrid_clock.cc:648] HybridClock initialized: now 1782018782037152 us; error 34 us; skew 500 ppm
Jun 21 05:13:02 dist-test-slave-vrq1 krb5kdc[17306](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782018782, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.131.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 05:13:02.040453 17681 init.cc:377] Logged in from keytab as kudu/127.3.131.131@KRBTEST.COM (short username kudu)
I20260621 05:13:02.041141 17681 webserver.cc:460] Webserver started at http://127.3.131.131:37631/ using document root <none> and password file <none>
I20260621 05:13:02.041412 17681 fs_manager.cc:359] Metadata directory not provided
I20260621 05:13:02.041487 17681 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 05:13:02.041651 17681 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 05:13:02.042761 17681 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782018616648578-3598-0/minicluster-data/ts-2/data/instance:
uuid: "0db2c38d6e21472abec748a4d51ff08b"
format_stamp: "Formatted at 2026-06-21 05:13:02 on dist-test-slave-vrq1"
server_key: "1b0bfccd06bec7c991568420c076d8f7"
server_key_iv: "395637a84e8143975246366fff991031"
server_key_version: "encryptionkey@0"
I20260621 05:13:02.043144 17681 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782018616648578-3598-0/minicluster-data/ts-2/wal/instance:
uuid: "0db2c38d6e21472abec748a4d51ff08b"
format_stamp: "Formatted at 2026-06-21 05:13:02 on dist-test-slave-vrq1"
server_key: "1b0bfccd06bec7c991568420c076d8f7"
server_key_iv: "395637a84e8143975246366fff991031"
server_key_version: "encryptionkey@0"
I20260621 05:13:02.044616 17681 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260621 05:13:02.045462 17696 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 05:13:02.045684 17681 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260621 05:13:02.045794 17681 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782018616648578-3598-0/minicluster-data/ts-2/data,/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782018616648578-3598-0/minicluster-data/ts-2/wal
uuid: "0db2c38d6e21472abec748a4d51ff08b"
format_stamp: "Formatted at 2026-06-21 05:13:02 on dist-test-slave-vrq1"
server_key: "1b0bfccd06bec7c991568420c076d8f7"
server_key_iv: "395637a84e8143975246366fff991031"
server_key_version: "encryptionkey@0"
I20260621 05:13:02.045881 17681 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782018616648578-3598-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782018616648578-3598-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782018616648578-3598-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 05:13:02.052644 17681 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 05:13:02.056108 17681 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 05:13:02.056280 17681 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 05:13:02.056532 17681 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 05:13:02.057036 17681 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 05:13:02.057080 17681 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 05:13:02.057104 17681 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 05:13:02.057117 17681 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 05:13:02.063922 17681 rpc_server.cc:307] RPC server started. Bound to: 127.3.131.131:33449
I20260621 05:13:02.064072 17809 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.131.131:33449 every 8 connection(s)
I20260621 05:13:02.064364 17681 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782018616648578-3598-0/minicluster-data/ts-2/data/info.pb
Jun 21 05:13:02 dist-test-slave-vrq1 krb5kdc[17306](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782018782, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.131.131@KRBTEST.COM for kudu/127.3.131.190@KRBTEST.COM
I20260621 05:13:02.074295 3598 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWZOx2_/build/release/bin/kudu as pid 17681
I20260621 05:13:02.074380 3598 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782018616648578-3598-0/minicluster-data/ts-2/wal/instance
I20260621 05:13:02.074579 3598 external_mini_cluster.cc:1428] Setting key 3121d6e72c94ede3bb7cae0aea5cf2dd
I20260621 05:13:02.076340 17391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:13:02.065443 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.131.131:33629 (local address 127.3.131.190:45579)
0621 05:13:02.065516 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0621 05:13:02.065518 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:13:02.065569 (+ 51us) server_negotiation.cc:407] Connection header received
0621 05:13:02.066232 (+ 663us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:13:02.066234 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:13:02.066250 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:13:02.066322 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:13:02.067795 (+ 1473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:13:02.068339 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:13:02.069439 (+ 1100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:13:02.069651 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:13:02.069719 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:13:02.072297 (+ 2578us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:13:02.072298 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:13:02.072299 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:13:02.072308 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:13:02.074000 (+ 1692us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:13:02.074621 (+ 621us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:13:02.074623 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:13:02.074624 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:13:02.074700 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:13:02.075035 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:13:02.075036 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:13:02.075036 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:13:02.075378 (+ 342us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:13:02.075475 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0621 05:13:02.076090 (+ 615us) server_negotiation.cc:299] Negotiation successful
0621 05:13:02.076219 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260621 05:13:02.076460 17812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:13:02.065168 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.131.190:45579 (local address 127.3.131.131:33629)
0621 05:13:02.065404 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0621 05:13:02.065458 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0621 05:13:02.066095 (+ 637us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:13:02.066382 (+ 287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:13:02.066386 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:13:02.067426 (+ 1040us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:13:02.067644 (+ 218us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:13:02.067650 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:13:02.068446 (+ 796us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:13:02.068447 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:13:02.069341 (+ 894us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:13:02.069343 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:13:02.069774 (+ 431us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:13:02.069775 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:13:02.069889 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:13:02.070535 (+ 646us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 05:13:02.070544 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:13:02.072163 (+ 1619us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 05:13:02.074143 (+ 1980us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:13:02.074156 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:13:02.074158 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:13:02.074530 (+ 372us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:13:02.074790 (+ 260us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:13:02.074791 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:13:02.074791 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:13:02.074919 (+ 128us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:13:02.075567 (+ 648us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 05:13:02.075576 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 05:13:02.075790 (+ 214us) client_negotiation.cc:769] Sending connection context
0621 05:13:02.075986 (+ 196us) client_negotiation.cc:240] Negotiation successful
0621 05:13:02.076223 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":151,"thread_start_us":80,"threads_started":1}
I20260621 05:13:02.077382 17810 heartbeater.cc:347] Connected to a master server at 127.3.131.190:45579
I20260621 05:13:02.077495 17810 heartbeater.cc:464] Registering TS with master...
I20260621 05:13:02.077718 17810 heartbeater.cc:511] Master 127.3.131.190:45579 requested a full tablet report, sending...
I20260621 05:13:02.078287 17351 ts_manager.cc:194] Registered new tserver with Master: 0db2c38d6e21472abec748a4d51ff08b (127.3.131.131:33449)
I20260621 05:13:02.078753 17351 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.131.131@KRBTEST.COM'} at 127.3.131.131:33629
I20260621 05:13:02.087953 3598 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260621 05:13:02.097086 17391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:13:02.089680 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39756 (local address 127.3.131.190:45579)
0621 05:13:02.089740 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0621 05:13:02.089742 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:13:02.089751 (+ 9us) server_negotiation.cc:407] Connection header received
0621 05:13:02.089780 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:13:02.089781 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:13:02.089795 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:13:02.089850 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:13:02.090614 (+ 764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:13:02.091342 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:13:02.092129 (+ 787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:13:02.092328 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:13:02.092390 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:13:02.093937 (+ 1547us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:13:02.093939 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:13:02.093939 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:13:02.093948 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:13:02.095597 (+ 1649us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:13:02.095992 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:13:02.095993 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:13:02.095993 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:13:02.096057 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:13:02.096430 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:13:02.096431 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:13:02.096431 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:13:02.096610 (+ 179us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:13:02.096704 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0621 05:13:02.096934 (+ 230us) server_negotiation.cc:299] Negotiation successful
0621 05:13:02.097012 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260621 05:13:02.099479 17351 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39756:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260621 05:13:02.101467 17351 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260621 05:13:02.109066 17824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:13:02.104382 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48650 (local address 127.3.131.129:45257)
0621 05:13:02.104674 (+ 292us) server_negotiation.cc:206] Beginning negotiation
0621 05:13:02.104680 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 05:13:02.104788 (+ 108us) server_negotiation.cc:407] Connection header received
0621 05:13:02.104898 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:13:02.104901 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:13:02.104917 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:13:02.105011 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 05:13:02.105213 (+ 202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:13:02.106371 (+ 1158us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:13:02.108001 (+ 1630us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:13:02.108619 (+ 618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:13:02.108658 (+ 39us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:13:02.108732 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0621 05:13:02.108947 (+ 215us) server_negotiation.cc:299] Negotiation successful
0621 05:13:02.108985 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":244,"thread_start_us":171,"threads_started":1}
I20260621 05:13:02.109056 17821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:13:02.104271 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.131.131:33449 (local address 127.0.0.1:34790)
0621 05:13:02.104916 (+ 645us) negotiation.cc:107] Waiting for socket to connect
0621 05:13:02.104924 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0621 05:13:02.104997 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:13:02.105425 (+ 428us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:13:02.105426 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:13:02.105428 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 05:13:02.105489 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:13:02.105491 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:13:02.106505 (+ 1014us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:13:02.106506 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:13:02.107656 (+ 1150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:13:02.107660 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:13:02.108717 (+ 1057us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:13:02.108718 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:13:02.108800 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:13:02.108814 (+ 14us) client_negotiation.cc:769] Sending connection context
0621 05:13:02.108887 (+ 73us) client_negotiation.cc:240] Negotiation successful
0621 05:13:02.108953 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":637,"mutex_wait_us":63,"thread_start_us":92,"threads_started":1}
I20260621 05:13:02.109197 17823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:13:02.104271 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.131.130:42965 (local address 127.0.0.1:58162)
0621 05:13:02.104767 (+ 496us) negotiation.cc:107] Waiting for socket to connect
0621 05:13:02.104780 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0621 05:13:02.104837 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:13:02.105340 (+ 503us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:13:02.105341 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:13:02.105343 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 05:13:02.105379 (+ 36us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:13:02.105380 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:13:02.106637 (+ 1257us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:13:02.106637 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:13:02.107893 (+ 1256us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:13:02.107896 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:13:02.108868 (+ 972us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:13:02.108869 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:13:02.108940 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:13:02.108945 (+ 5us) client_negotiation.cc:769] Sending connection context
0621 05:13:02.109005 (+ 60us) client_negotiation.cc:240] Negotiation successful
0621 05:13:02.109054 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":474,"mutex_wait_us":78,"thread_start_us":98,"threads_started":1}
I20260621 05:13:02.109318 17822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:13:02.104268 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.131.129:45257 (local address 127.0.0.1:48650)
0621 05:13:02.104664 (+ 396us) negotiation.cc:107] Waiting for socket to connect
0621 05:13:02.104707 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0621 05:13:02.104791 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:13:02.105073 (+ 282us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:13:02.105076 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:13:02.105079 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 05:13:02.105153 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:13:02.105156 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:13:02.106499 (+ 1343us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:13:02.106500 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:13:02.107899 (+ 1399us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:13:02.107901 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:13:02.108790 (+ 889us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:13:02.108790 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:13:02.108841 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:13:02.108845 (+ 4us) client_negotiation.cc:769] Sending connection context
0621 05:13:02.109167 (+ 322us) client_negotiation.cc:240] Negotiation successful
0621 05:13:02.109221 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":360,"mutex_wait_us":65,"thread_start_us":62,"threads_started":1}
I20260621 05:13:02.109472 17825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:13:02.104558 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58162 (local address 127.3.131.130:42965)
0621 05:13:02.105131 (+ 573us) server_negotiation.cc:206] Beginning negotiation
0621 05:13:02.105135 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 05:13:02.105146 (+ 11us) server_negotiation.cc:407] Connection header received
0621 05:13:02.105185 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:13:02.105187 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:13:02.105203 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:13:02.105289 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 05:13:02.105472 (+ 183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:13:02.106534 (+ 1062us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:13:02.107984 (+ 1450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:13:02.108748 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:13:02.108854 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:13:02.108927 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0621 05:13:02.109025 (+ 98us) server_negotiation.cc:299] Negotiation successful
0621 05:13:02.109070 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":536,"thread_start_us":83,"threads_started":1}
I20260621 05:13:02.109687 17826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:13:02.104550 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34790 (local address 127.3.131.131:33449)
0621 05:13:02.104769 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0621 05:13:02.104773 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 05:13:02.104996 (+ 223us) server_negotiation.cc:407] Connection header received
0621 05:13:02.105285 (+ 289us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:13:02.105289 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:13:02.105309 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:13:02.105420 (+ 111us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 05:13:02.105554 (+ 134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:13:02.106397 (+ 843us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:13:02.107760 (+ 1363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:13:02.108622 (+ 862us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:13:02.108707 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:13:02.108768 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0621 05:13:02.109319 (+ 551us) server_negotiation.cc:299] Negotiation successful
0621 05:13:02.109378 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":174,"thread_start_us":100,"threads_started":1}
I20260621 05:13:02.110134 17476 tablet_service.cc:1467] Processing CreateTablet for tablet d7c8b788a276447e9ef3bdb0a0a3a671 (DEFAULT_TABLE table=test-table [id=e75fc38419324aaaaac9542eb13967b1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 05:13:02.110236 17610 tablet_service.cc:1467] Processing CreateTablet for tablet d7c8b788a276447e9ef3bdb0a0a3a671 (DEFAULT_TABLE table=test-table [id=e75fc38419324aaaaac9542eb13967b1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 05:13:02.110230 17744 tablet_service.cc:1467] Processing CreateTablet for tablet d7c8b788a276447e9ef3bdb0a0a3a671 (DEFAULT_TABLE table=test-table [id=e75fc38419324aaaaac9542eb13967b1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 05:13:02.110560 17744 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d7c8b788a276447e9ef3bdb0a0a3a671. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 05:13:02.110555 17610 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d7c8b788a276447e9ef3bdb0a0a3a671. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 05:13:02.110556 17476 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d7c8b788a276447e9ef3bdb0a0a3a671. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 05:13:02.113236 17827 tablet_bootstrap.cc:492] T d7c8b788a276447e9ef3bdb0a0a3a671 P 0db2c38d6e21472abec748a4d51ff08b: Bootstrap starting.
I20260621 05:13:02.114040 17827 tablet_bootstrap.cc:654] T d7c8b788a276447e9ef3bdb0a0a3a671 P 0db2c38d6e21472abec748a4d51ff08b: Neither blocks nor log segments found. Creating new log.
I20260621 05:13:02.114130 17828 tablet_bootstrap.cc:492] T d7c8b788a276447e9ef3bdb0a0a3a671 P 828bb2f146144ed2826a65906ec39af5: Bootstrap starting.
I20260621 05:13:02.114462 17829 tablet_bootstrap.cc:492] T d7c8b788a276447e9ef3bdb0a0a3a671 P 090c4546eb4d4b02b3daa5bb23ec3444: Bootstrap starting.
I20260621 05:13:02.114843 17827 log.cc:824] T d7c8b788a276447e9ef3bdb0a0a3a671 P 0db2c38d6e21472abec748a4d51ff08b: Log is configured to *not* fsync() on all Append() calls
I20260621 05:13:02.114897 17828 tablet_bootstrap.cc:654] T d7c8b788a276447e9ef3bdb0a0a3a671 P 828bb2f146144ed2826a65906ec39af5: Neither blocks nor log segments found. Creating new log.
I20260621 05:13:02.115152 17829 tablet_bootstrap.cc:654] T d7c8b788a276447e9ef3bdb0a0a3a671 P 090c4546eb4d4b02b3daa5bb23ec3444: Neither blocks nor log segments found. Creating new log.
I20260621 05:13:02.115548 17828 log.cc:824] T d7c8b788a276447e9ef3bdb0a0a3a671 P 828bb2f146144ed2826a65906ec39af5: Log is configured to *not* fsync() on all Append() calls
I20260621 05:13:02.115842 17827 tablet_bootstrap.cc:492] T d7c8b788a276447e9ef3bdb0a0a3a671 P 0db2c38d6e21472abec748a4d51ff08b: No bootstrap required, opened a new log
I20260621 05:13:02.115864 17829 log.cc:824] T d7c8b788a276447e9ef3bdb0a0a3a671 P 090c4546eb4d4b02b3daa5bb23ec3444: Log is configured to *not* fsync() on all Append() calls
I20260621 05:13:02.115999 17827 ts_tablet_manager.cc:1397] T d7c8b788a276447e9ef3bdb0a0a3a671 P 0db2c38d6e21472abec748a4d51ff08b: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260621 05:13:02.116353 17828 tablet_bootstrap.cc:492] T d7c8b788a276447e9ef3bdb0a0a3a671 P 828bb2f146144ed2826a65906ec39af5: No bootstrap required, opened a new log
I20260621 05:13:02.116637 17828 ts_tablet_manager.cc:1397] T d7c8b788a276447e9ef3bdb0a0a3a671 P 828bb2f146144ed2826a65906ec39af5: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260621 05:13:02.116740 17829 tablet_bootstrap.cc:492] T d7c8b788a276447e9ef3bdb0a0a3a671 P 090c4546eb4d4b02b3daa5bb23ec3444: No bootstrap required, opened a new log
I20260621 05:13:02.116807 17829 ts_tablet_manager.cc:1397] T d7c8b788a276447e9ef3bdb0a0a3a671 P 090c4546eb4d4b02b3daa5bb23ec3444: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260621 05:13:02.118085 17827 raft_consensus.cc:348] T d7c8b788a276447e9ef3bdb0a0a3a671 P 0db2c38d6e21472abec748a4d51ff08b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "090c4546eb4d4b02b3daa5bb23ec3444" member_type: VOTER last_known_addr { host: "127.3.131.130" port: 42965 } } peers { permanent_uuid: "828bb2f146144ed2826a65906ec39af5" member_type: VOTER last_known_addr { host: "127.3.131.129" port: 45257 } } peers { permanent_uuid: "0db2c38d6e21472abec748a4d51ff08b" member_type: VOTER last_known_addr { host: "127.3.131.131" port: 33449 } }
I20260621 05:13:02.118287 17827 raft_consensus.cc:374] T d7c8b788a276447e9ef3bdb0a0a3a671 P 0db2c38d6e21472abec748a4d51ff08b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 05:13:02.118324 17827 raft_consensus.cc:729] T d7c8b788a276447e9ef3bdb0a0a3a671 P 0db2c38d6e21472abec748a4d51ff08b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0db2c38d6e21472abec748a4d51ff08b, State: Initialized, Role: FOLLOWER
I20260621 05:13:02.118456 17827 consensus_queue.cc:260] T d7c8b788a276447e9ef3bdb0a0a3a671 P 0db2c38d6e21472abec748a4d51ff08b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "090c4546eb4d4b02b3daa5bb23ec3444" member_type: VOTER last_known_addr { host: "127.3.131.130" port: 42965 } } peers { permanent_uuid: "828bb2f146144ed2826a65906ec39af5" member_type: VOTER last_known_addr { host: "127.3.131.129" port: 45257 } } peers { permanent_uuid: "0db2c38d6e21472abec748a4d51ff08b" member_type: VOTER last_known_addr { host: "127.3.131.131" port: 33449 } }
I20260621 05:13:02.118772 17827 ts_tablet_manager.cc:1428] T d7c8b788a276447e9ef3bdb0a0a3a671 P 0db2c38d6e21472abec748a4d51ff08b: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260621 05:13:02.118640 17829 raft_consensus.cc:348] T d7c8b788a276447e9ef3bdb0a0a3a671 P 090c4546eb4d4b02b3daa5bb23ec3444 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "090c4546eb4d4b02b3daa5bb23ec3444" member_type: VOTER last_known_addr { host: "127.3.131.130" port: 42965 } } peers { permanent_uuid: "828bb2f146144ed2826a65906ec39af5" member_type: VOTER last_known_addr { host: "127.3.131.129" port: 45257 } } peers { permanent_uuid: "0db2c38d6e21472abec748a4d51ff08b" member_type: VOTER last_known_addr { host: "127.3.131.131" port: 33449 } }
I20260621 05:13:02.118793 17810 heartbeater.cc:503] Master 127.3.131.190:45579 was elected leader, sending a full tablet report...
I20260621 05:13:02.118834 17829 raft_consensus.cc:374] T d7c8b788a276447e9ef3bdb0a0a3a671 P 090c4546eb4d4b02b3daa5bb23ec3444 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 05:13:02.118712 17828 raft_consensus.cc:348] T d7c8b788a276447e9ef3bdb0a0a3a671 P 828bb2f146144ed2826a65906ec39af5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "090c4546eb4d4b02b3daa5bb23ec3444" member_type: VOTER last_known_addr { host: "127.3.131.130" port: 42965 } } peers { permanent_uuid: "828bb2f146144ed2826a65906ec39af5" member_type: VOTER last_known_addr { host: "127.3.131.129" port: 45257 } } peers { permanent_uuid: "0db2c38d6e21472abec748a4d51ff08b" member_type: VOTER last_known_addr { host: "127.3.131.131" port: 33449 } }
I20260621 05:13:02.118870 17829 raft_consensus.cc:729] T d7c8b788a276447e9ef3bdb0a0a3a671 P 090c4546eb4d4b02b3daa5bb23ec3444 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 090c4546eb4d4b02b3daa5bb23ec3444, State: Initialized, Role: FOLLOWER
I20260621 05:13:02.118878 17828 raft_consensus.cc:374] T d7c8b788a276447e9ef3bdb0a0a3a671 P 828bb2f146144ed2826a65906ec39af5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 05:13:02.118911 17828 raft_consensus.cc:729] T d7c8b788a276447e9ef3bdb0a0a3a671 P 828bb2f146144ed2826a65906ec39af5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 828bb2f146144ed2826a65906ec39af5, State: Initialized, Role: FOLLOWER
I20260621 05:13:02.118983 17829 consensus_queue.cc:260] T d7c8b788a276447e9ef3bdb0a0a3a671 P 090c4546eb4d4b02b3daa5bb23ec3444 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "090c4546eb4d4b02b3daa5bb23ec3444" member_type: VOTER last_known_addr { host: "127.3.131.130" port: 42965 } } peers { permanent_uuid: "828bb2f146144ed2826a65906ec39af5" member_type: VOTER last_known_addr { host: "127.3.131.129" port: 45257 } } peers { permanent_uuid: "0db2c38d6e21472abec748a4d51ff08b" member_type: VOTER last_known_addr { host: "127.3.131.131" port: 33449 } }
I20260621 05:13:02.119019 17828 consensus_queue.cc:260] T d7c8b788a276447e9ef3bdb0a0a3a671 P 828bb2f146144ed2826a65906ec39af5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "090c4546eb4d4b02b3daa5bb23ec3444" member_type: VOTER last_known_addr { host: "127.3.131.130" port: 42965 } } peers { permanent_uuid: "828bb2f146144ed2826a65906ec39af5" member_type: VOTER last_known_addr { host: "127.3.131.129" port: 45257 } } peers { permanent_uuid: "0db2c38d6e21472abec748a4d51ff08b" member_type: VOTER last_known_addr { host: "127.3.131.131" port: 33449 } }
I20260621 05:13:02.119314 17828 ts_tablet_manager.cc:1428] T d7c8b788a276447e9ef3bdb0a0a3a671 P 828bb2f146144ed2826a65906ec39af5: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20260621 05:13:02.119303 17829 ts_tablet_manager.cc:1428] T d7c8b788a276447e9ef3bdb0a0a3a671 P 090c4546eb4d4b02b3daa5bb23ec3444: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260621 05:13:02.119485 17676 heartbeater.cc:503] Master 127.3.131.190:45579 was elected leader, sending a full tablet report...
I20260621 05:13:02.119827 17542 heartbeater.cc:503] Master 127.3.131.190:45579 was elected leader, sending a full tablet report...
W20260621 05:13:02.133272 17677 tablet.cc:2369] T d7c8b788a276447e9ef3bdb0a0a3a671 P 090c4546eb4d4b02b3daa5bb23ec3444: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260621 05:13:02.185376 17543 tablet.cc:2369] T d7c8b788a276447e9ef3bdb0a0a3a671 P 828bb2f146144ed2826a65906ec39af5: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260621 05:13:02.201573 17835 raft_consensus.cc:482] T d7c8b788a276447e9ef3bdb0a0a3a671 P 828bb2f146144ed2826a65906ec39af5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260621 05:13:02.201808 17835 raft_consensus.cc:504] T d7c8b788a276447e9ef3bdb0a0a3a671 P 828bb2f146144ed2826a65906ec39af5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "090c4546eb4d4b02b3daa5bb23ec3444" member_type: VOTER last_known_addr { host: "127.3.131.130" port: 42965 } } peers { permanent_uuid: "828bb2f146144ed2826a65906ec39af5" member_type: VOTER last_known_addr { host: "127.3.131.129" port: 45257 } } peers { permanent_uuid: "0db2c38d6e21472abec748a4d51ff08b" member_type: VOTER last_known_addr { host: "127.3.131.131" port: 33449 } }
I20260621 05:13:02.202266 17835 leader_election.cc:288] T d7c8b788a276447e9ef3bdb0a0a3a671 P 828bb2f146144ed2826a65906ec39af5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 090c4546eb4d4b02b3daa5bb23ec3444 (127.3.131.130:42965), 0db2c38d6e21472abec748a4d51ff08b (127.3.131.131:33449)
I20260621 05:13:02.205650 17837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:13:02.202448 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.131.130:42965 (local address 127.3.131.129:56175)
0621 05:13:02.202596 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0621 05:13:02.202608 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0621 05:13:02.202674 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:13:02.202837 (+ 163us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:13:02.202838 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:13:02.202846 (+ 8us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 05:13:02.202965 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:13:02.202968 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:13:02.203766 (+ 798us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:13:02.203767 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:13:02.204656 (+ 889us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:13:02.204658 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:13:02.205366 (+ 708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:13:02.205367 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:13:02.205452 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:13:02.205458 (+ 6us) client_negotiation.cc:769] Sending connection context
0621 05:13:02.205515 (+ 57us) client_negotiation.cc:240] Negotiation successful
0621 05:13:02.205563 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":131,"mutex_wait_us":8,"thread_start_us":64,"threads_started":1}
I20260621 05:13:02.205802 17825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:13:02.202506 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.131.129:56175 (local address 127.3.131.130:42965)
0621 05:13:02.202554 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0621 05:13:02.202556 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:13:02.202707 (+ 151us) server_negotiation.cc:407] Connection header received
0621 05:13:02.202746 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:13:02.202747 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:13:02.202758 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:13:02.202816 (+ 58us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 05:13:02.203084 (+ 268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:13:02.203684 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:13:02.204759 (+ 1075us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:13:02.205263 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:13:02.205325 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:13:02.205332 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0621 05:13:02.205561 (+ 229us) server_negotiation.cc:299] Negotiation successful
0621 05:13:02.205624 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260621 05:13:02.205888 17826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:13:02.202535 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.131.129:57209 (local address 127.3.131.131:33449)
0621 05:13:02.202581 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0621 05:13:02.202583 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:13:02.202783 (+ 200us) server_negotiation.cc:407] Connection header received
0621 05:13:02.202929 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:13:02.202930 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:13:02.202942 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:13:02.203003 (+ 61us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 05:13:02.203291 (+ 288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:13:02.203849 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:13:02.204915 (+ 1066us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:13:02.205402 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:13:02.205488 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:13:02.205498 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0621 05:13:02.205754 (+ 256us) server_negotiation.cc:299] Negotiation successful
0621 05:13:02.205808 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260621 05:13:02.206137 17836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:13:02.202452 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.131.131:33449 (local address 127.3.131.129:57209)
0621 05:13:02.202683 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0621 05:13:02.202698 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0621 05:13:02.202854 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:13:02.203068 (+ 214us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:13:02.203069 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:13:02.203072 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 05:13:02.203150 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:13:02.203152 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:13:02.203974 (+ 822us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:13:02.203975 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:13:02.204783 (+ 808us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:13:02.204785 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:13:02.205560 (+ 775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:13:02.205562 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:13:02.205648 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:13:02.205653 (+ 5us) client_negotiation.cc:769] Sending connection context
0621 05:13:02.205845 (+ 192us) client_negotiation.cc:240] Negotiation successful
0621 05:13:02.206029 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":71,"threads_started":1}
I20260621 05:13:02.206189 17630 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d7c8b788a276447e9ef3bdb0a0a3a671" candidate_uuid: "828bb2f146144ed2826a65906ec39af5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "090c4546eb4d4b02b3daa5bb23ec3444" is_pre_election: true
I20260621 05:13:02.206446 17630 raft_consensus.cc:2454] T d7c8b788a276447e9ef3bdb0a0a3a671 P 090c4546eb4d4b02b3daa5bb23ec3444 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 828bb2f146144ed2826a65906ec39af5 in term 0.
I20260621 05:13:02.206619 17764 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d7c8b788a276447e9ef3bdb0a0a3a671" candidate_uuid: "828bb2f146144ed2826a65906ec39af5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0db2c38d6e21472abec748a4d51ff08b" is_pre_election: true
I20260621 05:13:02.206697 17430 leader_election.cc:302] T d7c8b788a276447e9ef3bdb0a0a3a671 P 828bb2f146144ed2826a65906ec39af5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 090c4546eb4d4b02b3daa5bb23ec3444, 828bb2f146144ed2826a65906ec39af5; no voters:
I20260621 05:13:02.206812 17764 raft_consensus.cc:2454] T d7c8b788a276447e9ef3bdb0a0a3a671 P 0db2c38d6e21472abec748a4d51ff08b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 828bb2f146144ed2826a65906ec39af5 in term 0.
I20260621 05:13:02.206832 17835 raft_consensus.cc:2781] T d7c8b788a276447e9ef3bdb0a0a3a671 P 828bb2f146144ed2826a65906ec39af5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260621 05:13:02.206913 17835 raft_consensus.cc:482] T d7c8b788a276447e9ef3bdb0a0a3a671 P 828bb2f146144ed2826a65906ec39af5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260621 05:13:02.206954 17835 raft_consensus.cc:3037] T d7c8b788a276447e9ef3bdb0a0a3a671 P 828bb2f146144ed2826a65906ec39af5 [term 0 FOLLOWER]: Advancing to term 1
I20260621 05:13:02.207804 17835 raft_consensus.cc:504] T d7c8b788a276447e9ef3bdb0a0a3a671 P 828bb2f146144ed2826a65906ec39af5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "090c4546eb4d4b02b3daa5bb23ec3444" member_type: VOTER last_known_addr { host: "127.3.131.130" port: 42965 } } peers { permanent_uuid: "828bb2f146144ed2826a65906ec39af5" member_type: VOTER last_known_addr { host: "127.3.131.129" port: 45257 } } peers { permanent_uuid: "0db2c38d6e21472abec748a4d51ff08b" member_type: VOTER last_known_addr { host: "127.3.131.131" port: 33449 } }
I20260621 05:13:02.207983 17835 leader_election.cc:288] T d7c8b788a276447e9ef3bdb0a0a3a671 P 828bb2f146144ed2826a65906ec39af5 [CANDIDATE]: Term 1 election: Requested vote from peers 090c4546eb4d4b02b3daa5bb23ec3444 (127.3.131.130:42965), 0db2c38d6e21472abec748a4d51ff08b (127.3.131.131:33449)
I20260621 05:13:02.208207 17764 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d7c8b788a276447e9ef3bdb0a0a3a671" candidate_uuid: "828bb2f146144ed2826a65906ec39af5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0db2c38d6e21472abec748a4d51ff08b"
I20260621 05:13:02.208307 17764 raft_consensus.cc:3037] T d7c8b788a276447e9ef3bdb0a0a3a671 P 0db2c38d6e21472abec748a4d51ff08b [term 0 FOLLOWER]: Advancing to term 1
I20260621 05:13:02.208904 17630 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d7c8b788a276447e9ef3bdb0a0a3a671" candidate_uuid: "828bb2f146144ed2826a65906ec39af5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "090c4546eb4d4b02b3daa5bb23ec3444"
I20260621 05:13:02.209029 17630 raft_consensus.cc:3037] T d7c8b788a276447e9ef3bdb0a0a3a671 P 090c4546eb4d4b02b3daa5bb23ec3444 [term 0 FOLLOWER]: Advancing to term 1
I20260621 05:13:02.209314 17764 raft_consensus.cc:2454] T d7c8b788a276447e9ef3bdb0a0a3a671 P 0db2c38d6e21472abec748a4d51ff08b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 828bb2f146144ed2826a65906ec39af5 in term 1.
I20260621 05:13:02.209838 17630 raft_consensus.cc:2454] T d7c8b788a276447e9ef3bdb0a0a3a671 P 090c4546eb4d4b02b3daa5bb23ec3444 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 828bb2f146144ed2826a65906ec39af5 in term 1.
I20260621 05:13:02.210072 17432 leader_election.cc:302] T d7c8b788a276447e9ef3bdb0a0a3a671 P 828bb2f146144ed2826a65906ec39af5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0db2c38d6e21472abec748a4d51ff08b, 828bb2f146144ed2826a65906ec39af5; no voters:
I20260621 05:13:02.210220 17835 raft_consensus.cc:2781] T d7c8b788a276447e9ef3bdb0a0a3a671 P 828bb2f146144ed2826a65906ec39af5 [term 1 FOLLOWER]: Leader election won for term 1
I20260621 05:13:02.210438 17835 raft_consensus.cc:686] T d7c8b788a276447e9ef3bdb0a0a3a671 P 828bb2f146144ed2826a65906ec39af5 [term 1 LEADER]: Becoming Leader. State: Replica: 828bb2f146144ed2826a65906ec39af5, State: Running, Role: LEADER
I20260621 05:13:02.210525 17835 consensus_queue.cc:237] T d7c8b788a276447e9ef3bdb0a0a3a671 P 828bb2f146144ed2826a65906ec39af5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "090c4546eb4d4b02b3daa5bb23ec3444" member_type: VOTER last_known_addr { host: "127.3.131.130" port: 42965 } } peers { permanent_uuid: "828bb2f146144ed2826a65906ec39af5" member_type: VOTER last_known_addr { host: "127.3.131.129" port: 45257 } } peers { permanent_uuid: "0db2c38d6e21472abec748a4d51ff08b" member_type: VOTER last_known_addr { host: "127.3.131.131" port: 33449 } }
I20260621 05:13:02.211501 17349 catalog_manager.cc:5495] T d7c8b788a276447e9ef3bdb0a0a3a671 P 828bb2f146144ed2826a65906ec39af5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 828bb2f146144ed2826a65906ec39af5 (127.3.131.129). New cstate: current_term: 1 leader_uuid: "828bb2f146144ed2826a65906ec39af5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "090c4546eb4d4b02b3daa5bb23ec3444" member_type: VOTER last_known_addr { host: "127.3.131.130" port: 42965 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "828bb2f146144ed2826a65906ec39af5" member_type: VOTER last_known_addr { host: "127.3.131.129" port: 45257 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0db2c38d6e21472abec748a4d51ff08b" member_type: VOTER last_known_addr { host: "127.3.131.131" port: 33449 } health_report { overall_health: UNKNOWN } } }
I20260621 05:13:02.232034 17824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:13:02.228651 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48666 (local address 127.3.131.129:45257)
0621 05:13:02.228734 (+ 83us) server_negotiation.cc:206] Beginning negotiation
0621 05:13:02.228736 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:13:02.228743 (+ 7us) server_negotiation.cc:407] Connection header received
0621 05:13:02.228780 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:13:02.228780 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:13:02.228800 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:13:02.228879 (+ 79us) server_negotiation.cc:226] Negotiated authn=TOKEN
0621 05:13:02.229220 (+ 341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:13:02.229742 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:13:02.230749 (+ 1007us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:13:02.231156 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:13:02.231269 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:13:02.231487 (+ 218us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0621 05:13:02.231597 (+ 110us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0621 05:13:02.231667 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0621 05:13:02.231756 (+ 89us) server_negotiation.cc:299] Negotiation successful
0621 05:13:02.231779 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260621 05:13:02.234088 17764 raft_consensus.cc:1261] T d7c8b788a276447e9ef3bdb0a0a3a671 P 0db2c38d6e21472abec748a4d51ff08b [term 1 FOLLOWER]: Refusing update from remote peer 828bb2f146144ed2826a65906ec39af5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 05:13:02.234125 17630 raft_consensus.cc:1261] T d7c8b788a276447e9ef3bdb0a0a3a671 P 090c4546eb4d4b02b3daa5bb23ec3444 [term 1 FOLLOWER]: Refusing update from remote peer 828bb2f146144ed2826a65906ec39af5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 05:13:02.234536 17838 consensus_queue.cc:1035] T d7c8b788a276447e9ef3bdb0a0a3a671 P 828bb2f146144ed2826a65906ec39af5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "090c4546eb4d4b02b3daa5bb23ec3444" member_type: VOTER last_known_addr { host: "127.3.131.130" port: 42965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 05:13:02.234759 17835 consensus_queue.cc:1035] T d7c8b788a276447e9ef3bdb0a0a3a671 P 828bb2f146144ed2826a65906ec39af5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0db2c38d6e21472abec748a4d51ff08b" member_type: VOTER last_known_addr { host: "127.3.131.131" port: 33449 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 05:13:02.237635 17843 mvcc.cc:205] Tried to move back new op lower bound from 7299148932027420672 to 7299148931935039488. Current Snapshot: MvccSnapshot[applied={T|T < 7299148932027420672}]
I20260621 05:13:02.237524 17841 mvcc.cc:205] Tried to move back new op lower bound from 7299148932027420672 to 7299148931935039488. Current Snapshot: MvccSnapshot[applied={T|T < 7299148932027420672}]
I20260621 05:13:02.240808 17351 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39756:
table { table_name: "test-table" } modify_external_catalogs: true
I20260621 05:13:02.240953 17351 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39756:
table { table_name: "test-table" } modify_external_catalogs: true
I20260621 05:13:02.241770 17351 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f472f132d3144c2890a7a5b23ff3daca: Sending DeleteTablet for 3 replicas of tablet d7c8b788a276447e9ef3bdb0a0a3a671
I20260621 05:13:02.242184 17476 tablet_service.cc:1514] Processing DeleteTablet for tablet d7c8b788a276447e9ef3bdb0a0a3a671 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 05:13:02 UTC) from {username='kudu', principal='kudu/127.3.131.190@KRBTEST.COM'} at 127.0.0.1:48650
I20260621 05:13:02.242314 17744 tablet_service.cc:1514] Processing DeleteTablet for tablet d7c8b788a276447e9ef3bdb0a0a3a671 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 05:13:02 UTC) from {username='kudu', principal='kudu/127.3.131.190@KRBTEST.COM'} at 127.0.0.1:34790
I20260621 05:13:02.242439 17610 tablet_service.cc:1514] Processing DeleteTablet for tablet d7c8b788a276447e9ef3bdb0a0a3a671 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 05:13:02 UTC) from {username='kudu', principal='kudu/127.3.131.190@KRBTEST.COM'} at 127.0.0.1:58162
I20260621 05:13:02.242620 17852 tablet_replica.cc:330] T d7c8b788a276447e9ef3bdb0a0a3a671 P 0db2c38d6e21472abec748a4d51ff08b: stopping tablet replica
I20260621 05:13:02.242692 3598 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWZOx2_/build/release/bin/kudu with pid 17413
I20260621 05:13:02.242697 17853 tablet_replica.cc:330] T d7c8b788a276447e9ef3bdb0a0a3a671 P 090c4546eb4d4b02b3daa5bb23ec3444: stopping tablet replica
I20260621 05:13:02.242822 17852 raft_consensus.cc:2229] T d7c8b788a276447e9ef3bdb0a0a3a671 P 0db2c38d6e21472abec748a4d51ff08b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260621 05:13:02.243602 17853 raft_consensus.cc:2229] T d7c8b788a276447e9ef3bdb0a0a3a671 P 090c4546eb4d4b02b3daa5bb23ec3444 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260621 05:13:02.243758 17853 raft_consensus.cc:2258] T d7c8b788a276447e9ef3bdb0a0a3a671 P 090c4546eb4d4b02b3daa5bb23ec3444 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 05:13:02.244184 17853 ts_tablet_manager.cc:1905] T d7c8b788a276447e9ef3bdb0a0a3a671 P 090c4546eb4d4b02b3daa5bb23ec3444: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 05:13:02.244383 17852 raft_consensus.cc:2258] T d7c8b788a276447e9ef3bdb0a0a3a671 P 0db2c38d6e21472abec748a4d51ff08b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 05:13:02.244848 17852 ts_tablet_manager.cc:1905] T d7c8b788a276447e9ef3bdb0a0a3a671 P 0db2c38d6e21472abec748a4d51ff08b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 05:13:02.245728 17853 ts_tablet_manager.cc:1918] T d7c8b788a276447e9ef3bdb0a0a3a671 P 090c4546eb4d4b02b3daa5bb23ec3444: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260621 05:13:02.245821 17853 log.cc:1196] T d7c8b788a276447e9ef3bdb0a0a3a671 P 090c4546eb4d4b02b3daa5bb23ec3444: Deleting WAL directory at /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782018616648578-3598-0/minicluster-data/ts-1/wal/wals/d7c8b788a276447e9ef3bdb0a0a3a671
I20260621 05:13:02.246098 17853 ts_tablet_manager.cc:1939] T d7c8b788a276447e9ef3bdb0a0a3a671 P 090c4546eb4d4b02b3daa5bb23ec3444: Deleting consensus metadata
I20260621 05:13:02.246544 17336 catalog_manager.cc:4841] TS 090c4546eb4d4b02b3daa5bb23ec3444 (127.3.131.130:42965): tablet d7c8b788a276447e9ef3bdb0a0a3a671 (table test-table [id=e75fc38419324aaaaac9542eb13967b1]) successfully deleted
I20260621 05:13:02.246819 17852 ts_tablet_manager.cc:1918] T d7c8b788a276447e9ef3bdb0a0a3a671 P 0db2c38d6e21472abec748a4d51ff08b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260621 05:13:02.246907 17852 log.cc:1196] T d7c8b788a276447e9ef3bdb0a0a3a671 P 0db2c38d6e21472abec748a4d51ff08b: Deleting WAL directory at /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782018616648578-3598-0/minicluster-data/ts-2/wal/wals/d7c8b788a276447e9ef3bdb0a0a3a671
I20260621 05:13:02.247212 17852 ts_tablet_manager.cc:1939] T d7c8b788a276447e9ef3bdb0a0a3a671 P 0db2c38d6e21472abec748a4d51ff08b: Deleting consensus metadata
I20260621 05:13:02.247637 17338 catalog_manager.cc:4841] TS 0db2c38d6e21472abec748a4d51ff08b (127.3.131.131:33449): tablet d7c8b788a276447e9ef3bdb0a0a3a671 (table test-table [id=e75fc38419324aaaaac9542eb13967b1]) successfully deleted
W20260621 05:13:02.248905 17335 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.3.131.129:45257 (error 108)
I20260621 05:13:02.249205 3598 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWZOx2_/build/release/bin/kudu with pid 17547
I20260621 05:13:02.249846 17822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:13:02.249287 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.131.129:45257 (local address 127.0.0.1:48676)
0621 05:13:02.249752 (+ 465us) negotiation.cc:107] Waiting for socket to connect
0621 05:13:02.249802 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.131.129:45257: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":433}
W20260621 05:13:02.251322 17335 catalog_manager.cc:4584] TS 828bb2f146144ed2826a65906ec39af5 (127.3.131.129:45257): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet d7c8b788a276447e9ef3bdb0a0a3a671: Network error: Client connection negotiation failed: client connection to 127.3.131.129:45257: connect: Connection refused (error 111)
I20260621 05:13:02.254370 3598 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWZOx2_/build/release/bin/kudu with pid 17681
I20260621 05:13:02.259617 3598 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWZOx2_/build/release/bin/kudu with pid 17319
I20260621 05:13:02.265348 3598 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWZOx2_/build/release/bin/kudu
/tmp/dist-test-taskWZOx2_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782018616648578-3598-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782018616648578-3598-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782018616648578-3598-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782018616648578-3598-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.131.190:45579
--webserver_interface=127.3.131.190
--webserver_port=37647
--keytab_file=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782018616648578-3598-0/krb5kdc/kudu_127.3.131.190.keytab
--principal=kudu/127.3.131.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.131.148:41803
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.131.190:45579
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782018616648578-3598-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782018616648578-3598-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782018616648578-3598-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 05:13:02.355860 17855 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 05:13:02.356153 17855 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 05:13:02.356194 17855 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 05:13:02.357506 17855 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 05:13:02.357594 17855 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 05:13:02.357617 17855 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 05:13:02.357638 17855 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 05:13:02.357669 17855 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 05:13:02.359405 17855 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.131.148:41803
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782018616648578-3598-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782018616648578-3598-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.3.131.190:45579
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782018616648578-3598-0/krb5kdc/kudu_127.3.131.190.keytab
--principal=kudu/127.3.131.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.131.190:45579
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782018616648578-3598-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.131.190
--webserver_port=37647
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17855
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782018616648578-3598-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 21 Jun 2026 04:16:16 UTC on ccycloud.quasar-xycaha.root.comops.site
build id 4457881
I20260621 05:13:02.359722 17855 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 05:13:02.359974 17855 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 05:13:02.369513 17862 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:13:02.369587 17861 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:13:02.369655 17855 server_base.cc:1034] running on GCE node
W20260621 05:13:02.369693 17864 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:13:02.370198 17855 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 05:13:02.370648 17855 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 05:13:02.371873 17855 hybrid_clock.cc:648] HybridClock initialized: now 1782018782371852 us; error 41 us; skew 500 ppm
Jun 21 05:13:02 dist-test-slave-vrq1 krb5kdc[17306](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782018782, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.131.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 05:13:02.377928 17855 init.cc:377] Logged in from keytab as kudu/127.3.131.190@KRBTEST.COM (short username kudu)
I20260621 05:13:02.378484 17855 webserver.cc:460] Webserver started at http://127.3.131.190:37647/ using document root <none> and password file <none>
I20260621 05:13:02.378698 17855 fs_manager.cc:359] Metadata directory not provided
I20260621 05:13:02.378765 17855 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 05:13:02.380471 17855 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260621 05:13:02.381356 17871 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 05:13:02.381584 17855 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260621 05:13:02.381686 17855 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782018616648578-3598-0/minicluster-data/master-0/data,/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782018616648578-3598-0/minicluster-data/master-0/wal
uuid: "f472f132d3144c2890a7a5b23ff3daca"
format_stamp: "Formatted at 2026-06-21 05:13:01 on dist-test-slave-vrq1"
server_key: "f0397a2c76cb9696b49ac32a35ead456"
server_key_iv: "ce600c98bb104273486781bd45d2ef0d"
server_key_version: "encryptionkey@0"
I20260621 05:13:02.382135 17855 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782018616648578-3598-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782018616648578-3598-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782018616648578-3598-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 05:13:02.435652 17855 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 05:13:02.438187 17855 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 05:13:02.438321 17855 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 05:13:02.443519 17855 rpc_server.cc:307] RPC server started. Bound to: 127.3.131.190:45579
I20260621 05:13:02.443645 17923 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.131.190:45579 every 8 connection(s)
I20260621 05:13:02.444202 17855 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782018616648578-3598-0/minicluster-data/master-0/data/info.pb
I20260621 05:13:02.446044 17924 sys_catalog.cc:263] Verifying existing consensus state
I20260621 05:13:02.446692 17924 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f472f132d3144c2890a7a5b23ff3daca: Bootstrap starting.
I20260621 05:13:02.453047 3598 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWZOx2_/build/release/bin/kudu as pid 17855
I20260621 05:13:02.453504 3598 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWZOx2_/build/release/bin/kudu
/tmp/dist-test-taskWZOx2_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782018616648578-3598-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782018616648578-3598-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782018616648578-3598-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782018616648578-3598-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.131.129:45257
--local_ip_for_outbound_sockets=127.3.131.129
--tserver_master_addrs=127.3.131.190:45579
--webserver_port=45015
--webserver_interface=127.3.131.129
--keytab_file=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782018616648578-3598-0/krb5kdc/kudu_127.3.131.129.keytab
--principal=kudu/127.3.131.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.131.148:41803
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782018616648578-3598-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782018616648578-3598-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782018616648578-3598-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260621 05:13:02.456799 17924 log.cc:824] T 00000000000000000000000000000000 P f472f132d3144c2890a7a5b23ff3daca: Log is configured to *not* fsync() on all Append() calls
I20260621 05:13:02.459424 17924 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f472f132d3144c2890a7a5b23ff3daca: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260621 05:13:02.459775 17924 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f472f132d3144c2890a7a5b23ff3daca: Bootstrap complete.
I20260621 05:13:02.462257 17924 raft_consensus.cc:348] T 00000000000000000000000000000000 P f472f132d3144c2890a7a5b23ff3daca [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f472f132d3144c2890a7a5b23ff3daca" member_type: VOTER last_known_addr { host: "127.3.131.190" port: 45579 } }
I20260621 05:13:02.462563 17924 raft_consensus.cc:729] T 00000000000000000000000000000000 P f472f132d3144c2890a7a5b23ff3daca [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: f472f132d3144c2890a7a5b23ff3daca, State: Initialized, Role: FOLLOWER
I20260621 05:13:02.462775 17924 consensus_queue.cc:260] T 00000000000000000000000000000000 P f472f132d3144c2890a7a5b23ff3daca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f472f132d3144c2890a7a5b23ff3daca" member_type: VOTER last_known_addr { host: "127.3.131.190" port: 45579 } }
I20260621 05:13:02.462883 17924 raft_consensus.cc:388] T 00000000000000000000000000000000 P f472f132d3144c2890a7a5b23ff3daca [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260621 05:13:02.462945 17924 raft_consensus.cc:482] T 00000000000000000000000000000000 P f472f132d3144c2890a7a5b23ff3daca [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260621 05:13:02.462997 17924 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f472f132d3144c2890a7a5b23ff3daca [term 1 FOLLOWER]: Advancing to term 2
I20260621 05:13:02.464588 17924 raft_consensus.cc:504] T 00000000000000000000000000000000 P f472f132d3144c2890a7a5b23ff3daca [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f472f132d3144c2890a7a5b23ff3daca" member_type: VOTER last_known_addr { host: "127.3.131.190" port: 45579 } }
I20260621 05:13:02.464753 17924 leader_election.cc:302] T 00000000000000000000000000000000 P f472f132d3144c2890a7a5b23ff3daca [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f472f132d3144c2890a7a5b23ff3daca; no voters:
I20260621 05:13:02.464994 17924 leader_election.cc:288] T 00000000000000000000000000000000 P f472f132d3144c2890a7a5b23ff3daca [CANDIDATE]: Term 2 election: Requested vote from peers
I20260621 05:13:02.465080 17928 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f472f132d3144c2890a7a5b23ff3daca [term 2 FOLLOWER]: Leader election won for term 2
I20260621 05:13:02.465276 17928 raft_consensus.cc:686] T 00000000000000000000000000000000 P f472f132d3144c2890a7a5b23ff3daca [term 2 LEADER]: Becoming Leader. State: Replica: f472f132d3144c2890a7a5b23ff3daca, State: Running, Role: LEADER
I20260621 05:13:02.465301 17924 sys_catalog.cc:564] T 00000000000000000000000000000000 P f472f132d3144c2890a7a5b23ff3daca [sys.catalog]: configured and running, proceeding with master startup.
I20260621 05:13:02.465390 17928 consensus_queue.cc:237] T 00000000000000000000000000000000 P f472f132d3144c2890a7a5b23ff3daca [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f472f132d3144c2890a7a5b23ff3daca" member_type: VOTER last_known_addr { host: "127.3.131.190" port: 45579 } }
I20260621 05:13:02.465718 17928 sys_catalog.cc:455] T 00000000000000000000000000000000 P f472f132d3144c2890a7a5b23ff3daca [sys.catalog]: SysCatalogTable state changed. Reason: New leader f472f132d3144c2890a7a5b23ff3daca. Latest consensus state: current_term: 2 leader_uuid: "f472f132d3144c2890a7a5b23ff3daca" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f472f132d3144c2890a7a5b23ff3daca" member_type: VOTER last_known_addr { host: "127.3.131.190" port: 45579 } } }
I20260621 05:13:02.465849 17928 sys_catalog.cc:458] T 00000000000000000000000000000000 P f472f132d3144c2890a7a5b23ff3daca [sys.catalog]: This master's current role is: LEADER
I20260621 05:13:02.465719 17929 sys_catalog.cc:455] T 00000000000000000000000000000000 P f472f132d3144c2890a7a5b23ff3daca [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "f472f132d3144c2890a7a5b23ff3daca" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f472f132d3144c2890a7a5b23ff3daca" member_type: VOTER last_known_addr { host: "127.3.131.190" port: 45579 } } }
I20260621 05:13:02.466048 17929 sys_catalog.cc:458] T 00000000000000000000000000000000 P f472f132d3144c2890a7a5b23ff3daca [sys.catalog]: This master's current role is: LEADER
I20260621 05:13:02.466051 17935 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260621 05:13:02.467141 17935 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260621 05:13:02.467298 17935 catalog_manager.cc:1259] Loaded cluster ID: 01c16c36d7014e3d8bdcd707caf5e679
I20260621 05:13:02.467337 17935 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260621 05:13:02.467700 17935 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260621 05:13:02.472849 17935 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260621 05:13:02.475955 17935 catalog_manager.cc:1504] Loading token signing keys...
I20260621 05:13:02.476212 17935 catalog_manager.cc:5879] T 00000000000000000000000000000000 P f472f132d3144c2890a7a5b23ff3daca: Loaded TSK: 0
I20260621 05:13:02.476564 17935 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260621 05:13:02.549305 17926 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 05:13:02.549559 17926 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 05:13:02.549588 17926 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 05:13:02.550980 17926 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 05:13:02.551051 17926 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 05:13:02.551100 17926 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.131.129
I20260621 05:13:02.553045 17926 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.131.148:41803
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782018616648578-3598-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782018616648578-3598-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782018616648578-3598-0/krb5kdc/kudu_127.3.131.129.keytab
--principal=kudu/127.3.131.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.131.129:45257
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782018616648578-3598-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.131.129
--webserver_port=45015
--tserver_master_addrs=127.3.131.190:45579
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17926
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.131.129
--log_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782018616648578-3598-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 21 Jun 2026 04:16:16 UTC on ccycloud.quasar-xycaha.root.comops.site
build id 4457881
I20260621 05:13:02.553297 17926 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 05:13:02.553500 17926 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 05:13:02.561425 17952 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:13:02.561425 17951 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:13:02.561487 17954 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:13:02.562398 17926 server_base.cc:1034] running on GCE node
I20260621 05:13:02.562822 17926 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 05:13:02.563161 17926 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 05:13:02.564399 17926 hybrid_clock.cc:648] HybridClock initialized: now 1782018782564376 us; error 38 us; skew 500 ppm
Jun 21 05:13:02 dist-test-slave-vrq1 krb5kdc[17306](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782018782, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.131.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 05:13:02.567492 17926 init.cc:377] Logged in from keytab as kudu/127.3.131.129@KRBTEST.COM (short username kudu)
I20260621 05:13:02.568060 17926 webserver.cc:460] Webserver started at http://127.3.131.129:45015/ using document root <none> and password file <none>
I20260621 05:13:02.568301 17926 fs_manager.cc:359] Metadata directory not provided
I20260621 05:13:02.568364 17926 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 05:13:02.570746 17926 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.001s
I20260621 05:13:02.571718 17961 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 05:13:02.572382 17926 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260621 05:13:02.572510 17926 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782018616648578-3598-0/minicluster-data/ts-0/data,/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782018616648578-3598-0/minicluster-data/ts-0/wal
uuid: "828bb2f146144ed2826a65906ec39af5"
format_stamp: "Formatted at 2026-06-21 05:13:01 on dist-test-slave-vrq1"
server_key: "ef1e675f068b0c5dcdf64fe5a4bcbe36"
server_key_iv: "86e5551649e045a042c1f92215298a30"
server_key_version: "encryptionkey@0"
I20260621 05:13:02.573021 17926 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782018616648578-3598-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782018616648578-3598-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782018616648578-3598-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 05:13:02.599861 17926 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 05:13:02.602957 17926 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 05:13:02.603130 17926 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 05:13:02.603451 17926 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 05:13:02.603986 17968 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260621 05:13:02.605659 17926 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260621 05:13:02.605741 17926 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260621 05:13:02.605780 17926 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260621 05:13:02.606702 17926 ts_tablet_manager.cc:610] Registered 1 tablets
I20260621 05:13:02.606774 17926 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260621 05:13:02.607019 17968 tablet_bootstrap.cc:492] T d7c8b788a276447e9ef3bdb0a0a3a671 P 828bb2f146144ed2826a65906ec39af5: Bootstrap starting.
I20260621 05:13:02.614884 17926 rpc_server.cc:307] RPC server started. Bound to: 127.3.131.129:45257
I20260621 05:13:02.615481 17926 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782018616648578-3598-0/minicluster-data/ts-0/data/info.pb
I20260621 05:13:02.615718 18075 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.131.129:45257 every 8 connection(s)
I20260621 05:13:02.618063 17968 log.cc:824] T d7c8b788a276447e9ef3bdb0a0a3a671 P 828bb2f146144ed2826a65906ec39af5: Log is configured to *not* fsync() on all Append() calls
I20260621 05:13:02.620450 3598 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWZOx2_/build/release/bin/kudu as pid 17926
I20260621 05:13:02.620760 3598 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWZOx2_/build/release/bin/kudu
/tmp/dist-test-taskWZOx2_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782018616648578-3598-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782018616648578-3598-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782018616648578-3598-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782018616648578-3598-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.131.130:42965
--local_ip_for_outbound_sockets=127.3.131.130
--tserver_master_addrs=127.3.131.190:45579
--webserver_port=46151
--webserver_interface=127.3.131.130
--keytab_file=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782018616648578-3598-0/krb5kdc/kudu_127.3.131.130.keytab
--principal=kudu/127.3.131.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.131.148:41803
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782018616648578-3598-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782018616648578-3598-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782018616648578-3598-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260621 05:13:02.621207 17968 tablet_bootstrap.cc:492] T d7c8b788a276447e9ef3bdb0a0a3a671 P 828bb2f146144ed2826a65906ec39af5: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260621 05:13:02.621567 17968 tablet_bootstrap.cc:492] T d7c8b788a276447e9ef3bdb0a0a3a671 P 828bb2f146144ed2826a65906ec39af5: Bootstrap complete.
I20260621 05:13:02.622877 17968 ts_tablet_manager.cc:1397] T d7c8b788a276447e9ef3bdb0a0a3a671 P 828bb2f146144ed2826a65906ec39af5: Time spent bootstrapping tablet: real 0.016s user 0.003s sys 0.012s
I20260621 05:13:02.624390 17968 raft_consensus.cc:348] T d7c8b788a276447e9ef3bdb0a0a3a671 P 828bb2f146144ed2826a65906ec39af5 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "090c4546eb4d4b02b3daa5bb23ec3444" member_type: VOTER last_known_addr { host: "127.3.131.130" port: 42965 } } peers { permanent_uuid: "828bb2f146144ed2826a65906ec39af5" member_type: VOTER last_known_addr { host: "127.3.131.129" port: 45257 } } peers { permanent_uuid: "0db2c38d6e21472abec748a4d51ff08b" member_type: VOTER last_known_addr { host: "127.3.131.131" port: 33449 } }
I20260621 05:13:02.624704 17968 raft_consensus.cc:729] T d7c8b788a276447e9ef3bdb0a0a3a671 P 828bb2f146144ed2826a65906ec39af5 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 828bb2f146144ed2826a65906ec39af5, State: Initialized, Role: FOLLOWER
I20260621 05:13:02.624835 17968 consensus_queue.cc:260] T d7c8b788a276447e9ef3bdb0a0a3a671 P 828bb2f146144ed2826a65906ec39af5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "090c4546eb4d4b02b3daa5bb23ec3444" member_type: VOTER last_known_addr { host: "127.3.131.130" port: 42965 } } peers { permanent_uuid: "828bb2f146144ed2826a65906ec39af5" member_type: VOTER last_known_addr { host: "127.3.131.129" port: 45257 } } peers { permanent_uuid: "0db2c38d6e21472abec748a4d51ff08b" member_type: VOTER last_known_addr { host: "127.3.131.131" port: 33449 } }
I20260621 05:13:02.625116 17968 ts_tablet_manager.cc:1428] T d7c8b788a276447e9ef3bdb0a0a3a671 P 828bb2f146144ed2826a65906ec39af5: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
Jun 21 05:13:02 dist-test-slave-vrq1 krb5kdc[17306](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782018782, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.131.129@KRBTEST.COM for kudu/127.3.131.190@KRBTEST.COM
I20260621 05:13:02.632529 18079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:13:02.618887 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.131.190:45579 (local address 127.3.131.129:34965)
0621 05:13:02.619294 (+ 407us) negotiation.cc:107] Waiting for socket to connect
0621 05:13:02.619331 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0621 05:13:02.619875 (+ 544us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:13:02.620600 (+ 725us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:13:02.620605 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:13:02.621202 (+ 597us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:13:02.621563 (+ 361us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:13:02.621570 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:13:02.623076 (+ 1506us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:13:02.623076 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:13:02.624459 (+ 1383us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:13:02.624462 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:13:02.625410 (+ 948us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:13:02.625411 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:13:02.625558 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:13:02.626277 (+ 719us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 05:13:02.626288 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:13:02.628137 (+ 1849us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 05:13:02.630591 (+ 2454us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:13:02.630598 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:13:02.630601 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:13:02.630843 (+ 242us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:13:02.631268 (+ 425us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:13:02.631270 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:13:02.631270 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:13:02.631398 (+ 128us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:13:02.631967 (+ 569us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 05:13:02.631973 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 05:13:02.632112 (+ 139us) client_negotiation.cc:769] Sending connection context
0621 05:13:02.632204 (+ 92us) client_negotiation.cc:240] Negotiation successful
0621 05:13:02.632346 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":357,"thread_start_us":93,"threads_started":1}
I20260621 05:13:02.632722 18080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:13:02.618951 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.131.129:34965 (local address 127.3.131.190:45579)
0621 05:13:02.619145 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0621 05:13:02.619157 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0621 05:13:02.619443 (+ 286us) server_negotiation.cc:407] Connection header received
0621 05:13:02.620107 (+ 664us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:13:02.620112 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:13:02.620160 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:13:02.620384 (+ 224us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:13:02.621741 (+ 1357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:13:02.622917 (+ 1176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:13:02.624568 (+ 1651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:13:02.625190 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:13:02.625283 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:13:02.628262 (+ 2979us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:13:02.628268 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:13:02.628268 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:13:02.628284 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:13:02.630483 (+ 2199us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:13:02.630976 (+ 493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:13:02.630980 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:13:02.630981 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:13:02.631124 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:13:02.631489 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:13:02.631490 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:13:02.631490 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:13:02.631874 (+ 384us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:13:02.631973 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0621 05:13:02.632318 (+ 345us) server_negotiation.cc:299] Negotiation successful
0621 05:13:02.632484 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":148,"thread_start_us":78,"threads_started":1}
I20260621 05:13:02.633433 18076 heartbeater.cc:347] Connected to a master server at 127.3.131.190:45579
I20260621 05:13:02.633543 18076 heartbeater.cc:464] Registering TS with master...
I20260621 05:13:02.633733 18076 heartbeater.cc:511] Master 127.3.131.190:45579 requested a full tablet report, sending...
I20260621 05:13:02.634485 17888 ts_manager.cc:194] Registered new tserver with Master: 828bb2f146144ed2826a65906ec39af5 (127.3.131.129:45257)
I20260621 05:13:02.635578 17888 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.131.129@KRBTEST.COM'} at 127.3.131.129:34965
I20260621 05:13:02.636458 18076 heartbeater.cc:503] Master 127.3.131.190:45579 was elected leader, sending a full tablet report...
I20260621 05:13:02.643338 18083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:13:02.635084 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.131.129:45257 (local address 127.0.0.1:48692)
0621 05:13:02.639332 (+ 4248us) negotiation.cc:107] Waiting for socket to connect
0621 05:13:02.639380 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0621 05:13:02.639466 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:13:02.639696 (+ 230us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:13:02.639699 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:13:02.639702 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 05:13:02.639806 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:13:02.639808 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:13:02.641117 (+ 1309us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:13:02.641118 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:13:02.641951 (+ 833us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:13:02.641953 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:13:02.642945 (+ 992us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:13:02.642947 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:13:02.643055 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:13:02.643067 (+ 12us) client_negotiation.cc:769] Sending connection context
0621 05:13:02.643133 (+ 66us) client_negotiation.cc:240] Negotiation successful
0621 05:13:02.643243 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4203,"thread_start_us":226,"threads_started":1}
I20260621 05:13:02.643362 18084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:13:02.635335 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48692 (local address 127.3.131.129:45257)
0621 05:13:02.635534 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0621 05:13:02.635540 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 05:13:02.639475 (+ 3935us) server_negotiation.cc:407] Connection header received
0621 05:13:02.639540 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:13:02.639543 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:13:02.639563 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:13:02.639669 (+ 106us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 05:13:02.639889 (+ 220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:13:02.640979 (+ 1090us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:13:02.642040 (+ 1061us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:13:02.642800 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:13:02.642888 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:13:02.642946 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0621 05:13:02.643212 (+ 266us) server_negotiation.cc:299] Negotiation successful
0621 05:13:02.643276 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":164,"thread_start_us":111,"threads_started":1}
I20260621 05:13:02.643961 18008 tablet_service.cc:1514] Processing DeleteTablet for tablet d7c8b788a276447e9ef3bdb0a0a3a671 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 05:13:02 UTC) from {username='kudu', principal='kudu/127.3.131.190@KRBTEST.COM'} at 127.0.0.1:48692
I20260621 05:13:02.644635 18085 tablet_replica.cc:330] stopping tablet replica
I20260621 05:13:02.644750 18085 raft_consensus.cc:2229] T d7c8b788a276447e9ef3bdb0a0a3a671 P 828bb2f146144ed2826a65906ec39af5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260621 05:13:02.644887 18085 raft_consensus.cc:2258] T d7c8b788a276447e9ef3bdb0a0a3a671 P 828bb2f146144ed2826a65906ec39af5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 05:13:02.645190 18085 ts_tablet_manager.cc:1905] T d7c8b788a276447e9ef3bdb0a0a3a671 P 828bb2f146144ed2826a65906ec39af5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 05:13:02.647771 18085 ts_tablet_manager.cc:1918] T d7c8b788a276447e9ef3bdb0a0a3a671 P 828bb2f146144ed2826a65906ec39af5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260621 05:13:02.647892 18085 log.cc:1196] T d7c8b788a276447e9ef3bdb0a0a3a671 P 828bb2f146144ed2826a65906ec39af5: Deleting WAL directory at /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782018616648578-3598-0/minicluster-data/ts-0/wal/wals/d7c8b788a276447e9ef3bdb0a0a3a671
I20260621 05:13:02.648140 18085 ts_tablet_manager.cc:1939] T d7c8b788a276447e9ef3bdb0a0a3a671 P 828bb2f146144ed2826a65906ec39af5: Deleting consensus metadata
I20260621 05:13:02.648701 17872 catalog_manager.cc:4841] TS 828bb2f146144ed2826a65906ec39af5 (127.3.131.129:45257): tablet d7c8b788a276447e9ef3bdb0a0a3a671 (table test-table [id=e75fc38419324aaaaac9542eb13967b1]) successfully deleted
W20260621 05:13:02.714823 18081 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 05:13:02.715086 18081 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 05:13:02.715129 18081 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 05:13:02.716416 18081 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 05:13:02.716506 18081 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 05:13:02.716552 18081 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.131.130
I20260621 05:13:02.718127 18081 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.131.148:41803
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782018616648578-3598-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782018616648578-3598-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782018616648578-3598-0/krb5kdc/kudu_127.3.131.130.keytab
--principal=kudu/127.3.131.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.131.130:42965
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782018616648578-3598-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.131.130
--webserver_port=46151
--tserver_master_addrs=127.3.131.190:45579
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18081
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.131.130
--log_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782018616648578-3598-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 21 Jun 2026 04:16:16 UTC on ccycloud.quasar-xycaha.root.comops.site
build id 4457881
I20260621 05:13:02.718423 18081 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 05:13:02.718679 18081 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 05:13:02.724119 18093 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:13:02.724119 18090 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:13:02.724129 18091 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:13:02.724431 18081 server_base.cc:1034] running on GCE node
I20260621 05:13:02.724646 18081 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 05:13:02.724917 18081 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 05:13:02.726085 18081 hybrid_clock.cc:648] HybridClock initialized: now 1782018782726078 us; error 38 us; skew 500 ppm
Jun 21 05:13:02 dist-test-slave-vrq1 krb5kdc[17306](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782018782, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.131.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 05:13:02.728921 18081 init.cc:377] Logged in from keytab as kudu/127.3.131.130@KRBTEST.COM (short username kudu)
I20260621 05:13:02.729386 18081 webserver.cc:460] Webserver started at http://127.3.131.130:46151/ using document root <none> and password file <none>
I20260621 05:13:02.729646 18081 fs_manager.cc:359] Metadata directory not provided
I20260621 05:13:02.729707 18081 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 05:13:02.731329 18081 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260621 05:13:02.732096 18100 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 05:13:02.732415 18081 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260621 05:13:02.732545 18081 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782018616648578-3598-0/minicluster-data/ts-1/data,/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782018616648578-3598-0/minicluster-data/ts-1/wal
uuid: "090c4546eb4d4b02b3daa5bb23ec3444"
format_stamp: "Formatted at 2026-06-21 05:13:01 on dist-test-slave-vrq1"
server_key: "87ccbc83f2293f21959f5ce191249c7d"
server_key_iv: "766838c1befa0d8cc7fb0ec6b5f15826"
server_key_version: "encryptionkey@0"
I20260621 05:13:02.733074 18081 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782018616648578-3598-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782018616648578-3598-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782018616648578-3598-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 05:13:02.774194 18081 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 05:13:02.777086 18081 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 05:13:02.777276 18081 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 05:13:02.777590 18081 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 05:13:02.777938 18081 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 05:13:02.777993 18081 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 05:13:02.778036 18081 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 05:13:02.778062 18081 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 05:13:02.784088 18081 rpc_server.cc:307] RPC server started. Bound to: 127.3.131.130:42965
I20260621 05:13:02.784138 18213 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.131.130:42965 every 8 connection(s)
I20260621 05:13:02.784493 18081 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782018616648578-3598-0/minicluster-data/ts-1/data/info.pb
I20260621 05:13:02.789439 3598 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWZOx2_/build/release/bin/kudu as pid 18081
I20260621 05:13:02.789710 3598 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWZOx2_/build/release/bin/kudu
/tmp/dist-test-taskWZOx2_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782018616648578-3598-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782018616648578-3598-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782018616648578-3598-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782018616648578-3598-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.131.131:33449
--local_ip_for_outbound_sockets=127.3.131.131
--tserver_master_addrs=127.3.131.190:45579
--webserver_port=37631
--webserver_interface=127.3.131.131
--keytab_file=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782018616648578-3598-0/krb5kdc/kudu_127.3.131.131.keytab
--principal=kudu/127.3.131.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.131.148:41803
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782018616648578-3598-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782018616648578-3598-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782018616648578-3598-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 21 05:13:02 dist-test-slave-vrq1 krb5kdc[17306](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782018782, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.131.130@KRBTEST.COM for kudu/127.3.131.190@KRBTEST.COM
I20260621 05:13:02.795681 18080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:13:02.785501 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.131.130:59001 (local address 127.3.131.190:45579)
0621 05:13:02.785590 (+ 89us) server_negotiation.cc:206] Beginning negotiation
0621 05:13:02.785592 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:13:02.785799 (+ 207us) server_negotiation.cc:407] Connection header received
0621 05:13:02.786274 (+ 475us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:13:02.786276 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:13:02.786290 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:13:02.786341 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:13:02.787403 (+ 1062us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:13:02.787951 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:13:02.789074 (+ 1123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:13:02.789273 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:13:02.789361 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:13:02.791905 (+ 2544us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:13:02.791907 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:13:02.791907 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:13:02.791917 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:13:02.793792 (+ 1875us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:13:02.794280 (+ 488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:13:02.794281 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:13:02.794281 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:13:02.794354 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:13:02.794700 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:13:02.794701 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:13:02.794702 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:13:02.794896 (+ 194us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:13:02.794991 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0621 05:13:02.795404 (+ 413us) server_negotiation.cc:299] Negotiation successful
0621 05:13:02.795535 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260621 05:13:02.795710 18216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:13:02.785492 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.131.190:45579 (local address 127.3.131.130:59001)
0621 05:13:02.785647 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0621 05:13:02.785691 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0621 05:13:02.786180 (+ 489us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:13:02.786409 (+ 229us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:13:02.786413 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:13:02.787038 (+ 625us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:13:02.787306 (+ 268us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:13:02.787311 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:13:02.788044 (+ 733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:13:02.788045 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:13:02.789001 (+ 956us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:13:02.789004 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:13:02.789385 (+ 381us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:13:02.789386 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:13:02.789572 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:13:02.790160 (+ 588us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 05:13:02.790169 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:13:02.791737 (+ 1568us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 05:13:02.793913 (+ 2176us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:13:02.793922 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:13:02.793925 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:13:02.794194 (+ 269us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:13:02.794463 (+ 269us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:13:02.794464 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:13:02.794464 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:13:02.794598 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:13:02.795007 (+ 409us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 05:13:02.795013 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 05:13:02.795157 (+ 144us) client_negotiation.cc:769] Sending connection context
0621 05:13:02.795356 (+ 199us) client_negotiation.cc:240] Negotiation successful
0621 05:13:02.795528 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":85,"threads_started":1}
I20260621 05:13:02.796433 18214 heartbeater.cc:347] Connected to a master server at 127.3.131.190:45579
I20260621 05:13:02.796571 18214 heartbeater.cc:464] Registering TS with master...
I20260621 05:13:02.796814 18214 heartbeater.cc:511] Master 127.3.131.190:45579 requested a full tablet report, sending...
I20260621 05:13:02.797331 17888 ts_manager.cc:194] Registered new tserver with Master: 090c4546eb4d4b02b3daa5bb23ec3444 (127.3.131.130:42965)
I20260621 05:13:02.797847 17888 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.131.130@KRBTEST.COM'} at 127.3.131.130:59001
W20260621 05:13:02.876456 18217 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 05:13:02.876691 18217 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 05:13:02.876720 18217 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 05:13:02.878394 18217 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 05:13:02.878456 18217 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 05:13:02.878489 18217 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.131.131
I20260621 05:13:02.880263 18217 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.131.148:41803
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782018616648578-3598-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782018616648578-3598-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782018616648578-3598-0/krb5kdc/kudu_127.3.131.131.keytab
--principal=kudu/127.3.131.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.131.131:33449
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782018616648578-3598-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.131.131
--webserver_port=37631
--tserver_master_addrs=127.3.131.190:45579
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18217
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.131.131
--log_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782018616648578-3598-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 21 Jun 2026 04:16:16 UTC on ccycloud.quasar-xycaha.root.comops.site
build id 4457881
I20260621 05:13:02.880563 18217 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 05:13:02.880854 18217 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 05:13:02.886513 18225 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:13:02.886569 18223 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:13:02.886585 18222 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:13:02.886777 18217 server_base.cc:1034] running on GCE node
I20260621 05:13:02.887022 18217 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 05:13:02.887319 18217 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 05:13:02.888481 18217 hybrid_clock.cc:648] HybridClock initialized: now 1782018782888472 us; error 34 us; skew 500 ppm
Jun 21 05:13:02 dist-test-slave-vrq1 krb5kdc[17306](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782018782, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.131.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 05:13:02.891247 18217 init.cc:377] Logged in from keytab as kudu/127.3.131.131@KRBTEST.COM (short username kudu)
I20260621 05:13:02.891676 18217 webserver.cc:460] Webserver started at http://127.3.131.131:37631/ using document root <none> and password file <none>
I20260621 05:13:02.891873 18217 fs_manager.cc:359] Metadata directory not provided
I20260621 05:13:02.891927 18217 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 05:13:02.893322 18217 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260621 05:13:02.894006 18232 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 05:13:02.894248 18217 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260621 05:13:02.894340 18217 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782018616648578-3598-0/minicluster-data/ts-2/data,/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782018616648578-3598-0/minicluster-data/ts-2/wal
uuid: "0db2c38d6e21472abec748a4d51ff08b"
format_stamp: "Formatted at 2026-06-21 05:13:02 on dist-test-slave-vrq1"
server_key: "1b0bfccd06bec7c991568420c076d8f7"
server_key_iv: "395637a84e8143975246366fff991031"
server_key_version: "encryptionkey@0"
I20260621 05:13:02.894701 18217 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782018616648578-3598-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782018616648578-3598-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782018616648578-3598-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 05:13:02.910974 18217 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 05:13:02.913949 18217 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 05:13:02.914112 18217 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 05:13:02.914356 18217 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 05:13:02.914742 18217 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 05:13:02.914786 18217 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 05:13:02.914812 18217 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 05:13:02.914826 18217 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 05:13:02.921757 18217 rpc_server.cc:307] RPC server started. Bound to: 127.3.131.131:33449
I20260621 05:13:02.921820 18345 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.131.131:33449 every 8 connection(s)
I20260621 05:13:02.922176 18217 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782018616648578-3598-0/minicluster-data/ts-2/data/info.pb
I20260621 05:13:02.926246 3598 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWZOx2_/build/release/bin/kudu as pid 18217
Jun 21 05:13:02 dist-test-slave-vrq1 krb5kdc[17306](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782018782, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.131.131@KRBTEST.COM for kudu/127.3.131.190@KRBTEST.COM
I20260621 05:13:02.933084 18080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:13:02.923161 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.131.131:34683 (local address 127.3.131.190:45579)
0621 05:13:02.923287 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0621 05:13:02.923290 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0621 05:13:02.923487 (+ 197us) server_negotiation.cc:407] Connection header received
0621 05:13:02.923983 (+ 496us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:13:02.923985 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:13:02.923996 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:13:02.924058 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:13:02.925118 (+ 1060us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:13:02.925621 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:13:02.927009 (+ 1388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:13:02.927455 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:13:02.927642 (+ 187us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:13:02.929986 (+ 2344us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:13:02.929987 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:13:02.929988 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:13:02.929994 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:13:02.931557 (+ 1563us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:13:02.932015 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:13:02.932016 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:13:02.932017 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:13:02.932074 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:13:02.932370 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:13:02.932371 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:13:02.932372 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:13:02.932494 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:13:02.932569 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0621 05:13:02.932897 (+ 328us) server_negotiation.cc:299] Negotiation successful
0621 05:13:02.932989 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90}
I20260621 05:13:02.933336 18348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:13:02.923097 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.131.190:45579 (local address 127.3.131.131:34683)
0621 05:13:02.923353 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0621 05:13:02.923391 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0621 05:13:02.923868 (+ 477us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:13:02.924149 (+ 281us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:13:02.924152 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:13:02.924772 (+ 620us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:13:02.924969 (+ 197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:13:02.924974 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:13:02.925736 (+ 762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:13:02.925737 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:13:02.926925 (+ 1188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:13:02.926928 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:13:02.927641 (+ 713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:13:02.927642 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:13:02.927758 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:13:02.928418 (+ 660us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 05:13:02.928427 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:13:02.929885 (+ 1458us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 05:13:02.931645 (+ 1760us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:13:02.931653 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:13:02.931656 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:13:02.931904 (+ 248us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:13:02.932172 (+ 268us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:13:02.932172 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:13:02.932173 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:13:02.932296 (+ 123us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:13:02.932561 (+ 265us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 05:13:02.932568 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 05:13:02.932747 (+ 179us) client_negotiation.cc:769] Sending connection context
0621 05:13:02.932869 (+ 122us) client_negotiation.cc:240] Negotiation successful
0621 05:13:02.933046 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":208,"thread_start_us":165,"threads_started":1}
I20260621 05:13:02.933923 18346 heartbeater.cc:347] Connected to a master server at 127.3.131.190:45579
I20260621 05:13:02.934062 18346 heartbeater.cc:464] Registering TS with master...
I20260621 05:13:02.934262 18346 heartbeater.cc:511] Master 127.3.131.190:45579 requested a full tablet report, sending...
I20260621 05:13:02.934481 18350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:13:02.926695 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39760 (local address 127.3.131.190:45579)
0621 05:13:02.926827 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0621 05:13:02.926828 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0621 05:13:02.927081 (+ 253us) server_negotiation.cc:407] Connection header received
0621 05:13:02.927137 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:13:02.927137 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:13:02.927148 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:13:02.927259 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:13:02.928128 (+ 869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:13:02.928635 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:13:02.929390 (+ 755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:13:02.929663 (+ 273us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:13:02.929723 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:13:02.931333 (+ 1610us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:13:02.931334 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:13:02.931335 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:13:02.931342 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:13:02.932851 (+ 1509us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:13:02.933313 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:13:02.933314 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:13:02.933315 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:13:02.933380 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:13:02.933797 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:13:02.933798 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:13:02.933798 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:13:02.933932 (+ 134us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:13:02.934121 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0621 05:13:02.934188 (+ 67us) server_negotiation.cc:299] Negotiation successful
0621 05:13:02.934278 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":110,"thread_start_us":56,"threads_started":1}
I20260621 05:13:02.934783 17888 ts_manager.cc:194] Registered new tserver with Master: 0db2c38d6e21472abec748a4d51ff08b (127.3.131.131:33449)
I20260621 05:13:02.935137 17888 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.131.131@KRBTEST.COM'} at 127.3.131.131:34683
I20260621 05:13:02.936136 3598 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260621 05:13:02.939563 18350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:13:02.936712 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39772 (local address 127.3.131.190:45579)
0621 05:13:02.936784 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0621 05:13:02.936786 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:13:02.936893 (+ 107us) server_negotiation.cc:407] Connection header received
0621 05:13:02.936988 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:13:02.936989 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:13:02.937029 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:13:02.937113 (+ 84us) server_negotiation.cc:226] Negotiated authn=TOKEN
0621 05:13:02.937214 (+ 101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:13:02.937815 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:13:02.938651 (+ 836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:13:02.938871 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:13:02.938963 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:13:02.939141 (+ 178us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0621 05:13:02.939316 (+ 175us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0621 05:13:02.939409 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0621 05:13:02.939448 (+ 39us) server_negotiation.cc:299] Negotiation successful
0621 05:13:02.939476 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260621 05:13:02.939922 17888 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:39772:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260621 05:13:02.941574 17888 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260621 05:13:02.944038 18008 tablet_service.cc:1467] Processing CreateTablet for tablet 84b351425b4a43d68dacc18313dad243 (DEFAULT_TABLE table=test-table [id=50e5b31cf9c646528d1beb89aadcdaa6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 05:13:02.944314 18008 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 84b351425b4a43d68dacc18313dad243. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 05:13:02.945730 17968 tablet_bootstrap.cc:492] T 84b351425b4a43d68dacc18313dad243 P 828bb2f146144ed2826a65906ec39af5: Bootstrap starting.
I20260621 05:13:02.946151 17968 tablet_bootstrap.cc:654] T 84b351425b4a43d68dacc18313dad243 P 828bb2f146144ed2826a65906ec39af5: Neither blocks nor log segments found. Creating new log.
I20260621 05:13:02.946906 17968 tablet_bootstrap.cc:492] T 84b351425b4a43d68dacc18313dad243 P 828bb2f146144ed2826a65906ec39af5: No bootstrap required, opened a new log
I20260621 05:13:02.946962 17968 ts_tablet_manager.cc:1397] T 84b351425b4a43d68dacc18313dad243 P 828bb2f146144ed2826a65906ec39af5: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.000s
I20260621 05:13:02.947072 17968 raft_consensus.cc:348] T 84b351425b4a43d68dacc18313dad243 P 828bb2f146144ed2826a65906ec39af5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0db2c38d6e21472abec748a4d51ff08b" member_type: VOTER last_known_addr { host: "127.3.131.131" port: 33449 } } peers { permanent_uuid: "090c4546eb4d4b02b3daa5bb23ec3444" member_type: VOTER last_known_addr { host: "127.3.131.130" port: 42965 } } peers { permanent_uuid: "828bb2f146144ed2826a65906ec39af5" member_type: VOTER last_known_addr { host: "127.3.131.129" port: 45257 } }
I20260621 05:13:02.947146 17968 raft_consensus.cc:374] T 84b351425b4a43d68dacc18313dad243 P 828bb2f146144ed2826a65906ec39af5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 05:13:02.947216 17968 raft_consensus.cc:729] T 84b351425b4a43d68dacc18313dad243 P 828bb2f146144ed2826a65906ec39af5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 828bb2f146144ed2826a65906ec39af5, State: Initialized, Role: FOLLOWER
I20260621 05:13:02.947275 17968 consensus_queue.cc:260] T 84b351425b4a43d68dacc18313dad243 P 828bb2f146144ed2826a65906ec39af5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0db2c38d6e21472abec748a4d51ff08b" member_type: VOTER last_known_addr { host: "127.3.131.131" port: 33449 } } peers { permanent_uuid: "090c4546eb4d4b02b3daa5bb23ec3444" member_type: VOTER last_known_addr { host: "127.3.131.130" port: 42965 } } peers { permanent_uuid: "828bb2f146144ed2826a65906ec39af5" member_type: VOTER last_known_addr { host: "127.3.131.129" port: 45257 } }
I20260621 05:13:02.947369 17968 ts_tablet_manager.cc:1428] T 84b351425b4a43d68dacc18313dad243 P 828bb2f146144ed2826a65906ec39af5: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260621 05:13:02.950153 18354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:13:02.945697 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.131.130:42965 (local address 127.0.0.1:58176)
0621 05:13:02.945947 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0621 05:13:02.945958 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0621 05:13:02.946026 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:13:02.946314 (+ 288us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:13:02.946315 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:13:02.946318 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 05:13:02.946457 (+ 139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:13:02.946461 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:13:02.947942 (+ 1481us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:13:02.947943 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:13:02.949086 (+ 1143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:13:02.949089 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:13:02.949891 (+ 802us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:13:02.949892 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:13:02.949957 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:13:02.949963 (+ 6us) client_negotiation.cc:769] Sending connection context
0621 05:13:02.950022 (+ 59us) client_negotiation.cc:240] Negotiation successful
0621 05:13:02.950069 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":238,"thread_start_us":83,"threads_started":1}
I20260621 05:13:02.950163 18352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:13:02.943958 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58176 (local address 127.3.131.130:42965)
0621 05:13:02.944194 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0621 05:13:02.944199 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 05:13:02.946029 (+ 1830us) server_negotiation.cc:407] Connection header received
0621 05:13:02.946093 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:13:02.946096 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:13:02.946114 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:13:02.946211 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 05:13:02.946549 (+ 338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:13:02.947820 (+ 1271us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:13:02.949156 (+ 1336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:13:02.949796 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:13:02.949890 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:13:02.949955 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0621 05:13:02.950039 (+ 84us) server_negotiation.cc:299] Negotiation successful
0621 05:13:02.950078 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":191,"thread_start_us":122,"threads_started":1}
I20260621 05:13:02.950340 18083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:13:02.943880 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.131.131:33449 (local address 127.0.0.1:34798)
0621 05:13:02.944159 (+ 279us) negotiation.cc:107] Waiting for socket to connect
0621 05:13:02.944172 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0621 05:13:02.944217 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:13:02.946137 (+ 1920us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:13:02.946138 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:13:02.946139 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 05:13:02.946190 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:13:02.946191 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:13:02.947678 (+ 1487us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:13:02.947679 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:13:02.948948 (+ 1269us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:13:02.948951 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:13:02.950045 (+ 1094us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:13:02.950047 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:13:02.950116 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:13:02.950120 (+ 4us) client_negotiation.cc:769] Sending connection context
0621 05:13:02.950190 (+ 70us) client_negotiation.cc:240] Negotiation successful
0621 05:13:02.950246 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":258}
I20260621 05:13:02.950438 18353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:13:02.944036 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34798 (local address 127.3.131.131:33449)
0621 05:13:02.945947 (+ 1911us) server_negotiation.cc:206] Beginning negotiation
0621 05:13:02.945953 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 05:13:02.945965 (+ 12us) server_negotiation.cc:407] Connection header received
0621 05:13:02.946011 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:13:02.946014 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:13:02.946032 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:13:02.946139 (+ 107us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 05:13:02.946319 (+ 180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:13:02.947574 (+ 1255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:13:02.949057 (+ 1483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:13:02.949775 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:13:02.950031 (+ 256us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:13:02.950107 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0621 05:13:02.950206 (+ 99us) server_negotiation.cc:299] Negotiation successful
0621 05:13:02.950346 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1868,"thread_start_us":80,"threads_started":1}
I20260621 05:13:02.950920 18148 tablet_service.cc:1467] Processing CreateTablet for tablet 84b351425b4a43d68dacc18313dad243 (DEFAULT_TABLE table=test-table [id=50e5b31cf9c646528d1beb89aadcdaa6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 05:13:02.951077 18271 tablet_service.cc:1467] Processing CreateTablet for tablet 84b351425b4a43d68dacc18313dad243 (DEFAULT_TABLE table=test-table [id=50e5b31cf9c646528d1beb89aadcdaa6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 05:13:02.951361 18148 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 84b351425b4a43d68dacc18313dad243. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 05:13:02.951403 18271 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 84b351425b4a43d68dacc18313dad243. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 05:13:02.955610 18356 tablet_bootstrap.cc:492] T 84b351425b4a43d68dacc18313dad243 P 0db2c38d6e21472abec748a4d51ff08b: Bootstrap starting.
I20260621 05:13:02.955865 18357 tablet_bootstrap.cc:492] T 84b351425b4a43d68dacc18313dad243 P 090c4546eb4d4b02b3daa5bb23ec3444: Bootstrap starting.
I20260621 05:13:02.957012 18357 tablet_bootstrap.cc:654] T 84b351425b4a43d68dacc18313dad243 P 090c4546eb4d4b02b3daa5bb23ec3444: Neither blocks nor log segments found. Creating new log.
I20260621 05:13:02.957032 18356 tablet_bootstrap.cc:654] T 84b351425b4a43d68dacc18313dad243 P 0db2c38d6e21472abec748a4d51ff08b: Neither blocks nor log segments found. Creating new log.
I20260621 05:13:02.957656 18357 log.cc:824] T 84b351425b4a43d68dacc18313dad243 P 090c4546eb4d4b02b3daa5bb23ec3444: Log is configured to *not* fsync() on all Append() calls
I20260621 05:13:02.957656 18356 log.cc:824] T 84b351425b4a43d68dacc18313dad243 P 0db2c38d6e21472abec748a4d51ff08b: Log is configured to *not* fsync() on all Append() calls
I20260621 05:13:02.958649 18357 tablet_bootstrap.cc:492] T 84b351425b4a43d68dacc18313dad243 P 090c4546eb4d4b02b3daa5bb23ec3444: No bootstrap required, opened a new log
I20260621 05:13:02.958721 18357 ts_tablet_manager.cc:1397] T 84b351425b4a43d68dacc18313dad243 P 090c4546eb4d4b02b3daa5bb23ec3444: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260621 05:13:02.959481 18356 tablet_bootstrap.cc:492] T 84b351425b4a43d68dacc18313dad243 P 0db2c38d6e21472abec748a4d51ff08b: No bootstrap required, opened a new log
I20260621 05:13:02.959566 18356 ts_tablet_manager.cc:1397] T 84b351425b4a43d68dacc18313dad243 P 0db2c38d6e21472abec748a4d51ff08b: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260621 05:13:02.960942 18357 raft_consensus.cc:348] T 84b351425b4a43d68dacc18313dad243 P 090c4546eb4d4b02b3daa5bb23ec3444 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0db2c38d6e21472abec748a4d51ff08b" member_type: VOTER last_known_addr { host: "127.3.131.131" port: 33449 } } peers { permanent_uuid: "090c4546eb4d4b02b3daa5bb23ec3444" member_type: VOTER last_known_addr { host: "127.3.131.130" port: 42965 } } peers { permanent_uuid: "828bb2f146144ed2826a65906ec39af5" member_type: VOTER last_known_addr { host: "127.3.131.129" port: 45257 } }
I20260621 05:13:02.961143 18357 raft_consensus.cc:374] T 84b351425b4a43d68dacc18313dad243 P 090c4546eb4d4b02b3daa5bb23ec3444 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 05:13:02.961184 18357 raft_consensus.cc:729] T 84b351425b4a43d68dacc18313dad243 P 090c4546eb4d4b02b3daa5bb23ec3444 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 090c4546eb4d4b02b3daa5bb23ec3444, State: Initialized, Role: FOLLOWER
I20260621 05:13:02.961326 18357 consensus_queue.cc:260] T 84b351425b4a43d68dacc18313dad243 P 090c4546eb4d4b02b3daa5bb23ec3444 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0db2c38d6e21472abec748a4d51ff08b" member_type: VOTER last_known_addr { host: "127.3.131.131" port: 33449 } } peers { permanent_uuid: "090c4546eb4d4b02b3daa5bb23ec3444" member_type: VOTER last_known_addr { host: "127.3.131.130" port: 42965 } } peers { permanent_uuid: "828bb2f146144ed2826a65906ec39af5" member_type: VOTER last_known_addr { host: "127.3.131.129" port: 45257 } }
I20260621 05:13:02.961645 18357 ts_tablet_manager.cc:1428] T 84b351425b4a43d68dacc18313dad243 P 090c4546eb4d4b02b3daa5bb23ec3444: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260621 05:13:02.961664 18214 heartbeater.cc:503] Master 127.3.131.190:45579 was elected leader, sending a full tablet report...
I20260621 05:13:02.961576 18356 raft_consensus.cc:348] T 84b351425b4a43d68dacc18313dad243 P 0db2c38d6e21472abec748a4d51ff08b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0db2c38d6e21472abec748a4d51ff08b" member_type: VOTER last_known_addr { host: "127.3.131.131" port: 33449 } } peers { permanent_uuid: "090c4546eb4d4b02b3daa5bb23ec3444" member_type: VOTER last_known_addr { host: "127.3.131.130" port: 42965 } } peers { permanent_uuid: "828bb2f146144ed2826a65906ec39af5" member_type: VOTER last_known_addr { host: "127.3.131.129" port: 45257 } }
I20260621 05:13:02.961763 18356 raft_consensus.cc:374] T 84b351425b4a43d68dacc18313dad243 P 0db2c38d6e21472abec748a4d51ff08b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 05:13:02.961807 18356 raft_consensus.cc:729] T 84b351425b4a43d68dacc18313dad243 P 0db2c38d6e21472abec748a4d51ff08b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0db2c38d6e21472abec748a4d51ff08b, State: Initialized, Role: FOLLOWER
I20260621 05:13:02.961930 18356 consensus_queue.cc:260] T 84b351425b4a43d68dacc18313dad243 P 0db2c38d6e21472abec748a4d51ff08b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0db2c38d6e21472abec748a4d51ff08b" member_type: VOTER last_known_addr { host: "127.3.131.131" port: 33449 } } peers { permanent_uuid: "090c4546eb4d4b02b3daa5bb23ec3444" member_type: VOTER last_known_addr { host: "127.3.131.130" port: 42965 } } peers { permanent_uuid: "828bb2f146144ed2826a65906ec39af5" member_type: VOTER last_known_addr { host: "127.3.131.129" port: 45257 } }
I20260621 05:13:02.962246 18356 ts_tablet_manager.cc:1428] T 84b351425b4a43d68dacc18313dad243 P 0db2c38d6e21472abec748a4d51ff08b: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260621 05:13:02.962311 18346 heartbeater.cc:503] Master 127.3.131.190:45579 was elected leader, sending a full tablet report...
I20260621 05:13:03.008772 18360 raft_consensus.cc:482] T 84b351425b4a43d68dacc18313dad243 P 090c4546eb4d4b02b3daa5bb23ec3444 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260621 05:13:03.009008 18360 raft_consensus.cc:504] T 84b351425b4a43d68dacc18313dad243 P 090c4546eb4d4b02b3daa5bb23ec3444 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0db2c38d6e21472abec748a4d51ff08b" member_type: VOTER last_known_addr { host: "127.3.131.131" port: 33449 } } peers { permanent_uuid: "090c4546eb4d4b02b3daa5bb23ec3444" member_type: VOTER last_known_addr { host: "127.3.131.130" port: 42965 } } peers { permanent_uuid: "828bb2f146144ed2826a65906ec39af5" member_type: VOTER last_known_addr { host: "127.3.131.129" port: 45257 } }
I20260621 05:13:03.009438 18360 leader_election.cc:288] T 84b351425b4a43d68dacc18313dad243 P 090c4546eb4d4b02b3daa5bb23ec3444 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0db2c38d6e21472abec748a4d51ff08b (127.3.131.131:33449), 828bb2f146144ed2826a65906ec39af5 (127.3.131.129:45257)
I20260621 05:13:03.012594 18216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:13:03.009653 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.131.129:45257 (local address 127.3.131.130:41659)
0621 05:13:03.009700 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0621 05:13:03.009713 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0621 05:13:03.009757 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:13:03.010013 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:13:03.010015 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:13:03.010018 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 05:13:03.010099 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:13:03.010101 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:13:03.010847 (+ 746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:13:03.010848 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:13:03.011749 (+ 901us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:13:03.011752 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:13:03.012294 (+ 542us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:13:03.012295 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:13:03.012396 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:13:03.012400 (+ 4us) client_negotiation.cc:769] Sending connection context
0621 05:13:03.012452 (+ 52us) client_negotiation.cc:240] Negotiation successful
0621 05:13:03.012486 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":27,"mutex_wait_us":54}
I20260621 05:13:03.012614 18084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:13:03.009742 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.131.130:41659 (local address 127.3.131.129:45257)
0621 05:13:03.009786 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0621 05:13:03.009788 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:13:03.009799 (+ 11us) server_negotiation.cc:407] Connection header received
0621 05:13:03.009833 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:13:03.009834 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:13:03.009845 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:13:03.009926 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 05:13:03.010176 (+ 250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:13:03.010756 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:13:03.011843 (+ 1087us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:13:03.012214 (+ 371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:13:03.012272 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:13:03.012279 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0621 05:13:03.012481 (+ 202us) server_negotiation.cc:299] Negotiation successful
0621 05:13:03.012538 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260621 05:13:03.012663 18353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:13:03.009739 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.131.130:38369 (local address 127.3.131.131:33449)
0621 05:13:03.009786 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0621 05:13:03.009788 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:13:03.010007 (+ 219us) server_negotiation.cc:407] Connection header received
0621 05:13:03.010039 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:13:03.010040 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:13:03.010051 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:13:03.010124 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 05:13:03.010315 (+ 191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:13:03.010767 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:13:03.011891 (+ 1124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:13:03.012288 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:13:03.012359 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:13:03.012369 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0621 05:13:03.012540 (+ 171us) server_negotiation.cc:299] Negotiation successful
0621 05:13:03.012587 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260621 05:13:03.012708 18362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:13:03.009654 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.131.131:33449 (local address 127.3.131.130:38369)
0621 05:13:03.009882 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0621 05:13:03.009901 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0621 05:13:03.009963 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:13:03.010151 (+ 188us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:13:03.010152 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:13:03.010154 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 05:13:03.010230 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:13:03.010231 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:13:03.010881 (+ 650us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:13:03.010882 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:13:03.011760 (+ 878us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:13:03.011762 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:13:03.012371 (+ 609us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:13:03.012372 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:13:03.012434 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:13:03.012455 (+ 21us) client_negotiation.cc:769] Sending connection context
0621 05:13:03.012516 (+ 61us) client_negotiation.cc:240] Negotiation successful
0621 05:13:03.012552 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":207,"mutex_wait_us":74,"thread_start_us":111,"threads_started":1}
I20260621 05:13:03.012917 18026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84b351425b4a43d68dacc18313dad243" candidate_uuid: "090c4546eb4d4b02b3daa5bb23ec3444" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "828bb2f146144ed2826a65906ec39af5" is_pre_election: true
I20260621 05:13:03.013068 18026 raft_consensus.cc:2454] T 84b351425b4a43d68dacc18313dad243 P 828bb2f146144ed2826a65906ec39af5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 090c4546eb4d4b02b3daa5bb23ec3444 in term 0.
I20260621 05:13:03.013157 18299 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84b351425b4a43d68dacc18313dad243" candidate_uuid: "090c4546eb4d4b02b3daa5bb23ec3444" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0db2c38d6e21472abec748a4d51ff08b" is_pre_election: true
I20260621 05:13:03.013332 18299 raft_consensus.cc:2454] T 84b351425b4a43d68dacc18313dad243 P 0db2c38d6e21472abec748a4d51ff08b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 090c4546eb4d4b02b3daa5bb23ec3444 in term 0.
I20260621 05:13:03.013319 18101 leader_election.cc:302] T 84b351425b4a43d68dacc18313dad243 P 090c4546eb4d4b02b3daa5bb23ec3444 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 090c4546eb4d4b02b3daa5bb23ec3444, 828bb2f146144ed2826a65906ec39af5; no voters:
I20260621 05:13:03.013468 18360 raft_consensus.cc:2781] T 84b351425b4a43d68dacc18313dad243 P 090c4546eb4d4b02b3daa5bb23ec3444 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260621 05:13:03.013540 18360 raft_consensus.cc:482] T 84b351425b4a43d68dacc18313dad243 P 090c4546eb4d4b02b3daa5bb23ec3444 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260621 05:13:03.013568 18360 raft_consensus.cc:3037] T 84b351425b4a43d68dacc18313dad243 P 090c4546eb4d4b02b3daa5bb23ec3444 [term 0 FOLLOWER]: Advancing to term 1
I20260621 05:13:03.014333 18360 raft_consensus.cc:504] T 84b351425b4a43d68dacc18313dad243 P 090c4546eb4d4b02b3daa5bb23ec3444 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0db2c38d6e21472abec748a4d51ff08b" member_type: VOTER last_known_addr { host: "127.3.131.131" port: 33449 } } peers { permanent_uuid: "090c4546eb4d4b02b3daa5bb23ec3444" member_type: VOTER last_known_addr { host: "127.3.131.130" port: 42965 } } peers { permanent_uuid: "828bb2f146144ed2826a65906ec39af5" member_type: VOTER last_known_addr { host: "127.3.131.129" port: 45257 } }
I20260621 05:13:03.014495 18360 leader_election.cc:288] T 84b351425b4a43d68dacc18313dad243 P 090c4546eb4d4b02b3daa5bb23ec3444 [CANDIDATE]: Term 1 election: Requested vote from peers 0db2c38d6e21472abec748a4d51ff08b (127.3.131.131:33449), 828bb2f146144ed2826a65906ec39af5 (127.3.131.129:45257)
I20260621 05:13:03.014675 18299 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84b351425b4a43d68dacc18313dad243" candidate_uuid: "090c4546eb4d4b02b3daa5bb23ec3444" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0db2c38d6e21472abec748a4d51ff08b"
I20260621 05:13:03.014678 18026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84b351425b4a43d68dacc18313dad243" candidate_uuid: "090c4546eb4d4b02b3daa5bb23ec3444" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "828bb2f146144ed2826a65906ec39af5"
I20260621 05:13:03.014770 18299 raft_consensus.cc:3037] T 84b351425b4a43d68dacc18313dad243 P 0db2c38d6e21472abec748a4d51ff08b [term 0 FOLLOWER]: Advancing to term 1
I20260621 05:13:03.014770 18026 raft_consensus.cc:3037] T 84b351425b4a43d68dacc18313dad243 P 828bb2f146144ed2826a65906ec39af5 [term 0 FOLLOWER]: Advancing to term 1
I20260621 05:13:03.015614 18299 raft_consensus.cc:2454] T 84b351425b4a43d68dacc18313dad243 P 0db2c38d6e21472abec748a4d51ff08b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 090c4546eb4d4b02b3daa5bb23ec3444 in term 1.
I20260621 05:13:03.015897 18104 leader_election.cc:302] T 84b351425b4a43d68dacc18313dad243 P 090c4546eb4d4b02b3daa5bb23ec3444 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 090c4546eb4d4b02b3daa5bb23ec3444, 0db2c38d6e21472abec748a4d51ff08b; no voters:
I20260621 05:13:03.016039 18026 raft_consensus.cc:2454] T 84b351425b4a43d68dacc18313dad243 P 828bb2f146144ed2826a65906ec39af5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 090c4546eb4d4b02b3daa5bb23ec3444 in term 1.
I20260621 05:13:03.016032 18360 raft_consensus.cc:2781] T 84b351425b4a43d68dacc18313dad243 P 090c4546eb4d4b02b3daa5bb23ec3444 [term 1 FOLLOWER]: Leader election won for term 1
I20260621 05:13:03.016199 18360 raft_consensus.cc:686] T 84b351425b4a43d68dacc18313dad243 P 090c4546eb4d4b02b3daa5bb23ec3444 [term 1 LEADER]: Becoming Leader. State: Replica: 090c4546eb4d4b02b3daa5bb23ec3444, State: Running, Role: LEADER
I20260621 05:13:03.016316 18360 consensus_queue.cc:237] T 84b351425b4a43d68dacc18313dad243 P 090c4546eb4d4b02b3daa5bb23ec3444 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0db2c38d6e21472abec748a4d51ff08b" member_type: VOTER last_known_addr { host: "127.3.131.131" port: 33449 } } peers { permanent_uuid: "090c4546eb4d4b02b3daa5bb23ec3444" member_type: VOTER last_known_addr { host: "127.3.131.130" port: 42965 } } peers { permanent_uuid: "828bb2f146144ed2826a65906ec39af5" member_type: VOTER last_known_addr { host: "127.3.131.129" port: 45257 } }
I20260621 05:13:03.017223 17887 catalog_manager.cc:5495] T 84b351425b4a43d68dacc18313dad243 P 090c4546eb4d4b02b3daa5bb23ec3444 reported cstate change: term changed from 0 to 1, leader changed from <none> to 090c4546eb4d4b02b3daa5bb23ec3444 (127.3.131.130). New cstate: current_term: 1 leader_uuid: "090c4546eb4d4b02b3daa5bb23ec3444" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0db2c38d6e21472abec748a4d51ff08b" member_type: VOTER last_known_addr { host: "127.3.131.131" port: 33449 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "090c4546eb4d4b02b3daa5bb23ec3444" member_type: VOTER last_known_addr { host: "127.3.131.130" port: 42965 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "828bb2f146144ed2826a65906ec39af5" member_type: VOTER last_known_addr { host: "127.3.131.129" port: 45257 } health_report { overall_health: UNKNOWN } } }
W20260621 05:13:03.035161 18215 tablet.cc:2369] T 84b351425b4a43d68dacc18313dad243 P 090c4546eb4d4b02b3daa5bb23ec3444: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260621 05:13:03.042189 18352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:13:03.039624 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58178 (local address 127.3.131.130:42965)
0621 05:13:03.039696 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0621 05:13:03.039697 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0621 05:13:03.039706 (+ 9us) server_negotiation.cc:407] Connection header received
0621 05:13:03.039748 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:13:03.039750 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:13:03.039767 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:13:03.039827 (+ 60us) server_negotiation.cc:226] Negotiated authn=TOKEN
0621 05:13:03.040003 (+ 176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:13:03.040485 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:13:03.041364 (+ 879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:13:03.041574 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:13:03.041630 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:13:03.041820 (+ 190us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0621 05:13:03.041925 (+ 105us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0621 05:13:03.041987 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0621 05:13:03.042089 (+ 102us) server_negotiation.cc:299] Negotiation successful
0621 05:13:03.042107 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260621 05:13:03.044258 18299 raft_consensus.cc:1261] T 84b351425b4a43d68dacc18313dad243 P 0db2c38d6e21472abec748a4d51ff08b [term 1 FOLLOWER]: Refusing update from remote peer 090c4546eb4d4b02b3daa5bb23ec3444: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 05:13:03.044272 18026 raft_consensus.cc:1261] T 84b351425b4a43d68dacc18313dad243 P 828bb2f146144ed2826a65906ec39af5 [term 1 FOLLOWER]: Refusing update from remote peer 090c4546eb4d4b02b3daa5bb23ec3444: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 05:13:03.044699 18363 consensus_queue.cc:1035] T 84b351425b4a43d68dacc18313dad243 P 090c4546eb4d4b02b3daa5bb23ec3444 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0db2c38d6e21472abec748a4d51ff08b" member_type: VOTER last_known_addr { host: "127.3.131.131" port: 33449 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 05:13:03.044859 18360 consensus_queue.cc:1035] T 84b351425b4a43d68dacc18313dad243 P 090c4546eb4d4b02b3daa5bb23ec3444 [LEADER]: Connected to new peer: Peer: permanent_uuid: "828bb2f146144ed2826a65906ec39af5" member_type: VOTER last_known_addr { host: "127.3.131.129" port: 45257 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 05:13:03.047542 18366 mvcc.cc:205] Tried to move back new op lower bound from 7299148935346298880 to 7299148935235649536. Current Snapshot: MvccSnapshot[applied={T|T < 7299148935346298880}]
I20260621 05:13:03.047698 18368 mvcc.cc:205] Tried to move back new op lower bound from 7299148935346298880 to 7299148935235649536. Current Snapshot: MvccSnapshot[applied={T|T < 7299148935346298880}]
I20260621 05:13:03.051113 17887 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:39772:
table { table_name: "test-table" } modify_external_catalogs: true
I20260621 05:13:03.051458 17887 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:39772:
table { table_name: "test-table" } modify_external_catalogs: true
I20260621 05:13:03.052475 17887 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f472f132d3144c2890a7a5b23ff3daca: Sending DeleteTablet for 3 replicas of tablet 84b351425b4a43d68dacc18313dad243
I20260621 05:13:03.052960 3598 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWZOx2_/build/release/bin/kudu with pid 17926
I20260621 05:13:03.052999 18148 tablet_service.cc:1514] Processing DeleteTablet for tablet 84b351425b4a43d68dacc18313dad243 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 05:13:03 UTC) from {username='kudu', principal='kudu/127.3.131.190@KRBTEST.COM'} at 127.0.0.1:58176
I20260621 05:13:03.053574 18271 tablet_service.cc:1514] Processing DeleteTablet for tablet 84b351425b4a43d68dacc18313dad243 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 05:13:03 UTC) from {username='kudu', principal='kudu/127.3.131.190@KRBTEST.COM'} at 127.0.0.1:34798
I20260621 05:13:03.057116 18377 tablet_replica.cc:330] T 84b351425b4a43d68dacc18313dad243 P 0db2c38d6e21472abec748a4d51ff08b: stopping tablet replica
I20260621 05:13:03.057350 18376 tablet_replica.cc:330] T 84b351425b4a43d68dacc18313dad243 P 090c4546eb4d4b02b3daa5bb23ec3444: stopping tablet replica
I20260621 05:13:03.057569 18376 raft_consensus.cc:2229] T 84b351425b4a43d68dacc18313dad243 P 090c4546eb4d4b02b3daa5bb23ec3444 [term 1 LEADER]: Raft consensus shutting down.
I20260621 05:13:03.057781 18377 raft_consensus.cc:2229] T 84b351425b4a43d68dacc18313dad243 P 0db2c38d6e21472abec748a4d51ff08b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260621 05:13:03.058032 18376 raft_consensus.cc:2258] T 84b351425b4a43d68dacc18313dad243 P 090c4546eb4d4b02b3daa5bb23ec3444 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 05:13:03.058228 18377 raft_consensus.cc:2258] T 84b351425b4a43d68dacc18313dad243 P 0db2c38d6e21472abec748a4d51ff08b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 05:13:03.059082 18377 ts_tablet_manager.cc:1905] T 84b351425b4a43d68dacc18313dad243 P 0db2c38d6e21472abec748a4d51ff08b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 05:13:03.059088 18376 ts_tablet_manager.cc:1905] T 84b351425b4a43d68dacc18313dad243 P 090c4546eb4d4b02b3daa5bb23ec3444: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 05:13:03.061091 18376 ts_tablet_manager.cc:1918] T 84b351425b4a43d68dacc18313dad243 P 090c4546eb4d4b02b3daa5bb23ec3444: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260621 05:13:03.061196 18376 log.cc:1196] T 84b351425b4a43d68dacc18313dad243 P 090c4546eb4d4b02b3daa5bb23ec3444: Deleting WAL directory at /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782018616648578-3598-0/minicluster-data/ts-1/wal/wals/84b351425b4a43d68dacc18313dad243
I20260621 05:13:03.061399 18377 ts_tablet_manager.cc:1918] T 84b351425b4a43d68dacc18313dad243 P 0db2c38d6e21472abec748a4d51ff08b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260621 05:13:03.061522 18376 ts_tablet_manager.cc:1939] T 84b351425b4a43d68dacc18313dad243 P 090c4546eb4d4b02b3daa5bb23ec3444: Deleting consensus metadata
I20260621 05:13:03.061616 18377 log.cc:1196] T 84b351425b4a43d68dacc18313dad243 P 0db2c38d6e21472abec748a4d51ff08b: Deleting WAL directory at /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782018616648578-3598-0/minicluster-data/ts-2/wal/wals/84b351425b4a43d68dacc18313dad243
I20260621 05:13:03.062068 18377 ts_tablet_manager.cc:1939] T 84b351425b4a43d68dacc18313dad243 P 0db2c38d6e21472abec748a4d51ff08b: Deleting consensus metadata
I20260621 05:13:03.062049 17873 catalog_manager.cc:4841] TS 090c4546eb4d4b02b3daa5bb23ec3444 (127.3.131.130:42965): tablet 84b351425b4a43d68dacc18313dad243 (table test-table [id=50e5b31cf9c646528d1beb89aadcdaa6]) successfully deleted
W20260621 05:13:03.062382 17872 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.3.131.129:45257 (error 108)
I20260621 05:13:03.062568 17875 catalog_manager.cc:4841] TS 0db2c38d6e21472abec748a4d51ff08b (127.3.131.131:33449): tablet 84b351425b4a43d68dacc18313dad243 (table test-table [id=50e5b31cf9c646528d1beb89aadcdaa6]) successfully deleted
I20260621 05:13:03.062970 18083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:13:03.062802 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.131.129:45257 (local address 127.0.0.1:48698)
0621 05:13:03.062875 (+ 73us) negotiation.cc:107] Waiting for socket to connect
0621 05:13:03.062928 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.131.129:45257: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260621 05:13:03.063161 3598 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWZOx2_/build/release/bin/kudu with pid 18081
W20260621 05:13:03.063110 17872 catalog_manager.cc:4584] TS 828bb2f146144ed2826a65906ec39af5 (127.3.131.129:45257): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 84b351425b4a43d68dacc18313dad243: Network error: Client connection negotiation failed: client connection to 127.3.131.129:45257: connect: Connection refused (error 111)
I20260621 05:13:03.069314 3598 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWZOx2_/build/release/bin/kudu with pid 18217
I20260621 05:13:03.074653 3598 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWZOx2_/build/release/bin/kudu with pid 17855
I20260621 05:13:03.080317 3598 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWZOx2_/build/release/bin/kudu
/tmp/dist-test-taskWZOx2_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782018616648578-3598-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782018616648578-3598-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782018616648578-3598-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782018616648578-3598-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.131.190:45579
--webserver_interface=127.3.131.190
--webserver_port=37647
--keytab_file=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782018616648578-3598-0/krb5kdc/kudu_127.3.131.190.keytab
--principal=kudu/127.3.131.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.131.148:41803
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.131.190:45579
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782018616648578-3598-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782018616648578-3598-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782018616648578-3598-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 05:13:03.165897 18379 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 05:13:03.166170 18379 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 05:13:03.166213 18379 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 05:13:03.167517 18379 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 05:13:03.167599 18379 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 05:13:03.167624 18379 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 05:13:03.167660 18379 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 05:13:03.167707 18379 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 05:13:03.169327 18379 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.131.148:41803
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782018616648578-3598-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782018616648578-3598-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.131.190:45579
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782018616648578-3598-0/krb5kdc/kudu_127.3.131.190.keytab
--principal=kudu/127.3.131.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.131.190:45579
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782018616648578-3598-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.131.190
--webserver_port=37647
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18379
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782018616648578-3598-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 21 Jun 2026 04:16:16 UTC on ccycloud.quasar-xycaha.root.comops.site
build id 4457881
I20260621 05:13:03.169653 18379 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 05:13:03.169900 18379 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 05:13:03.175303 18384 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:13:03.175339 18385 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:13:03.175354 18379 server_base.cc:1034] running on GCE node
W20260621 05:13:03.175315 18387 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:13:03.175851 18379 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 05:13:03.176229 18379 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 05:13:03.177433 18379 hybrid_clock.cc:648] HybridClock initialized: now 1782018783177415 us; error 34 us; skew 500 ppm
Jun 21 05:13:03 dist-test-slave-vrq1 krb5kdc[17306](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782018783, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.131.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 05:13:03.180018 18379 init.cc:377] Logged in from keytab as kudu/127.3.131.190@KRBTEST.COM (short username kudu)
I20260621 05:13:03.180472 18379 webserver.cc:460] Webserver started at http://127.3.131.190:37647/ using document root <none> and password file <none>
I20260621 05:13:03.180737 18379 fs_manager.cc:359] Metadata directory not provided
I20260621 05:13:03.180788 18379 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 05:13:03.182209 18379 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260621 05:13:03.183076 18394 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 05:13:03.183398 18379 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260621 05:13:03.183490 18379 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782018616648578-3598-0/minicluster-data/master-0/data,/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782018616648578-3598-0/minicluster-data/master-0/wal
uuid: "f472f132d3144c2890a7a5b23ff3daca"
format_stamp: "Formatted at 2026-06-21 05:13:01 on dist-test-slave-vrq1"
server_key: "f0397a2c76cb9696b49ac32a35ead456"
server_key_iv: "ce600c98bb104273486781bd45d2ef0d"
server_key_version: "encryptionkey@0"
I20260621 05:13:03.184073 18379 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782018616648578-3598-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782018616648578-3598-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782018616648578-3598-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 05:13:03.198146 18379 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 05:13:03.200750 18379 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 05:13:03.200922 18379 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 05:13:03.205382 18379 rpc_server.cc:307] RPC server started. Bound to: 127.3.131.190:45579
I20260621 05:13:03.205541 18446 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.131.190:45579 every 8 connection(s)
I20260621 05:13:03.205991 18379 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782018616648578-3598-0/minicluster-data/master-0/data/info.pb
I20260621 05:13:03.207718 18447 sys_catalog.cc:263] Verifying existing consensus state
I20260621 05:13:03.208432 18447 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f472f132d3144c2890a7a5b23ff3daca: Bootstrap starting.
I20260621 05:13:03.215885 3598 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWZOx2_/build/release/bin/kudu as pid 18379
I20260621 05:13:03.216248 3598 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWZOx2_/build/release/bin/kudu
/tmp/dist-test-taskWZOx2_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782018616648578-3598-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782018616648578-3598-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782018616648578-3598-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782018616648578-3598-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.131.129:45257
--local_ip_for_outbound_sockets=127.3.131.129
--tserver_master_addrs=127.3.131.190:45579
--webserver_port=45015
--webserver_interface=127.3.131.129
--keytab_file=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782018616648578-3598-0/krb5kdc/kudu_127.3.131.129.keytab
--principal=kudu/127.3.131.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.131.148:41803
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782018616648578-3598-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782018616648578-3598-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782018616648578-3598-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260621 05:13:03.217397 18447 log.cc:824] T 00000000000000000000000000000000 P f472f132d3144c2890a7a5b23ff3daca: Log is configured to *not* fsync() on all Append() calls
I20260621 05:13:03.220396 18447 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f472f132d3144c2890a7a5b23ff3daca: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260621 05:13:03.220738 18447 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f472f132d3144c2890a7a5b23ff3daca: Bootstrap complete.
I20260621 05:13:03.223224 18447 raft_consensus.cc:348] T 00000000000000000000000000000000 P f472f132d3144c2890a7a5b23ff3daca [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f472f132d3144c2890a7a5b23ff3daca" member_type: VOTER last_known_addr { host: "127.3.131.190" port: 45579 } }
I20260621 05:13:03.223531 18447 raft_consensus.cc:729] T 00000000000000000000000000000000 P f472f132d3144c2890a7a5b23ff3daca [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: f472f132d3144c2890a7a5b23ff3daca, State: Initialized, Role: FOLLOWER
I20260621 05:13:03.223743 18447 consensus_queue.cc:260] T 00000000000000000000000000000000 P f472f132d3144c2890a7a5b23ff3daca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f472f132d3144c2890a7a5b23ff3daca" member_type: VOTER last_known_addr { host: "127.3.131.190" port: 45579 } }
I20260621 05:13:03.223851 18447 raft_consensus.cc:388] T 00000000000000000000000000000000 P f472f132d3144c2890a7a5b23ff3daca [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260621 05:13:03.223905 18447 raft_consensus.cc:482] T 00000000000000000000000000000000 P f472f132d3144c2890a7a5b23ff3daca [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260621 05:13:03.223955 18447 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f472f132d3144c2890a7a5b23ff3daca [term 2 FOLLOWER]: Advancing to term 3
I20260621 05:13:03.225067 18447 raft_consensus.cc:504] T 00000000000000000000000000000000 P f472f132d3144c2890a7a5b23ff3daca [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f472f132d3144c2890a7a5b23ff3daca" member_type: VOTER last_known_addr { host: "127.3.131.190" port: 45579 } }
I20260621 05:13:03.225198 18447 leader_election.cc:302] T 00000000000000000000000000000000 P f472f132d3144c2890a7a5b23ff3daca [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f472f132d3144c2890a7a5b23ff3daca; no voters:
I20260621 05:13:03.225417 18447 leader_election.cc:288] T 00000000000000000000000000000000 P f472f132d3144c2890a7a5b23ff3daca [CANDIDATE]: Term 3 election: Requested vote from peers
I20260621 05:13:03.225457 18451 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f472f132d3144c2890a7a5b23ff3daca [term 3 FOLLOWER]: Leader election won for term 3
I20260621 05:13:03.225566 18451 raft_consensus.cc:686] T 00000000000000000000000000000000 P f472f132d3144c2890a7a5b23ff3daca [term 3 LEADER]: Becoming Leader. State: Replica: f472f132d3144c2890a7a5b23ff3daca, State: Running, Role: LEADER
I20260621 05:13:03.225660 18451 consensus_queue.cc:237] T 00000000000000000000000000000000 P f472f132d3144c2890a7a5b23ff3daca [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f472f132d3144c2890a7a5b23ff3daca" member_type: VOTER last_known_addr { host: "127.3.131.190" port: 45579 } }
I20260621 05:13:03.225730 18447 sys_catalog.cc:564] T 00000000000000000000000000000000 P f472f132d3144c2890a7a5b23ff3daca [sys.catalog]: configured and running, proceeding with master startup.
I20260621 05:13:03.225946 18451 sys_catalog.cc:455] T 00000000000000000000000000000000 P f472f132d3144c2890a7a5b23ff3daca [sys.catalog]: SysCatalogTable state changed. Reason: New leader f472f132d3144c2890a7a5b23ff3daca. Latest consensus state: current_term: 3 leader_uuid: "f472f132d3144c2890a7a5b23ff3daca" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f472f132d3144c2890a7a5b23ff3daca" member_type: VOTER last_known_addr { host: "127.3.131.190" port: 45579 } } }
I20260621 05:13:03.226023 18451 sys_catalog.cc:458] T 00000000000000000000000000000000 P f472f132d3144c2890a7a5b23ff3daca [sys.catalog]: This master's current role is: LEADER
I20260621 05:13:03.225950 18452 sys_catalog.cc:455] T 00000000000000000000000000000000 P f472f132d3144c2890a7a5b23ff3daca [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "f472f132d3144c2890a7a5b23ff3daca" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f472f132d3144c2890a7a5b23ff3daca" member_type: VOTER last_known_addr { host: "127.3.131.190" port: 45579 } } }
I20260621 05:13:03.226082 18452 sys_catalog.cc:458] T 00000000000000000000000000000000 P f472f132d3144c2890a7a5b23ff3daca [sys.catalog]: This master's current role is: LEADER
I20260621 05:13:03.226712 18455 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260621 05:13:03.227809 18455 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260621 05:13:03.227968 18455 catalog_manager.cc:1259] Loaded cluster ID: 01c16c36d7014e3d8bdcd707caf5e679
I20260621 05:13:03.228097 18455 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260621 05:13:03.228554 18455 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7fce1f2eb9ad google::LogMessage::Fail() at ??:0
@ 0x7fce1f2ee88a google::LogMessage::SendToLog() at ??:0
@ 0x7fce1f2eb4de google::LogMessage::Flush() at ??:0
@ 0x7fce1f2eca19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7fce2720c0f1 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_.isra.591 at ??:0
@ 0x7fce272158b9 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7fce211b90d3 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7fce211ae954 kudu::Thread::SuperviseThread() at ??:0
@ 0x7fce220126db start_thread at ??:0
@ 0x7fce1dde961f clone at ??:0
W20260621 05:13:03.324510 18449 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 05:13:03.324714 18449 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 05:13:03.324740 18449 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 05:13:03.325960 18449 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 05:13:03.326004 18449 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 05:13:03.326031 18449 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.131.129
I20260621 05:13:03.327602 18449 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.131.148:41803
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782018616648578-3598-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782018616648578-3598-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782018616648578-3598-0/krb5kdc/kudu_127.3.131.129.keytab
--principal=kudu/127.3.131.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.131.129:45257
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782018616648578-3598-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.131.129
--webserver_port=45015
--tserver_master_addrs=127.3.131.190:45579
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18449
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.131.129
--log_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782018616648578-3598-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 21 Jun 2026 04:16:16 UTC on ccycloud.quasar-xycaha.root.comops.site
build id 4457881
I20260621 05:13:03.327863 18449 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 05:13:03.328125 18449 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260621 05:13:03.333580 18449 server_base.cc:1034] running on GCE node
W20260621 05:13:03.333598 18483 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:13:03.333715 18484 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:13:03.333858 18486 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:13:03.334092 18449 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 05:13:03.334367 18449 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 05:13:03.335553 18449 hybrid_clock.cc:648] HybridClock initialized: now 1782018783335535 us; error 36 us; skew 500 ppm
Jun 21 05:13:03 dist-test-slave-vrq1 krb5kdc[17306](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782018783, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.131.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 05:13:03.338152 18449 init.cc:377] Logged in from keytab as kudu/127.3.131.129@KRBTEST.COM (short username kudu)
I20260621 05:13:03.338663 18449 webserver.cc:460] Webserver started at http://127.3.131.129:45015/ using document root <none> and password file <none>
I20260621 05:13:03.338923 18449 fs_manager.cc:359] Metadata directory not provided
I20260621 05:13:03.339004 18449 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 05:13:03.340466 18449 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260621 05:13:03.341288 18493 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 05:13:03.341507 18449 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260621 05:13:03.341643 18449 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782018616648578-3598-0/minicluster-data/ts-0/data,/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782018616648578-3598-0/minicluster-data/ts-0/wal
uuid: "828bb2f146144ed2826a65906ec39af5"
format_stamp: "Formatted at 2026-06-21 05:13:01 on dist-test-slave-vrq1"
server_key: "ef1e675f068b0c5dcdf64fe5a4bcbe36"
server_key_iv: "86e5551649e045a042c1f92215298a30"
server_key_version: "encryptionkey@0"
I20260621 05:13:03.342059 18449 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782018616648578-3598-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782018616648578-3598-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782018616648578-3598-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 05:13:03.381007 18449 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 05:13:03.383694 18449 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 05:13:03.383888 18449 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 05:13:03.384169 18449 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 05:13:03.384690 18500 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260621 05:13:03.386969 18449 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260621 05:13:03.387070 18449 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.003s user 0.000s sys 0.000s
I20260621 05:13:03.387130 18449 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260621 05:13:03.388016 18449 ts_tablet_manager.cc:610] Registered 1 tablets
I20260621 05:13:03.388077 18449 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260621 05:13:03.388145 18500 tablet_bootstrap.cc:492] T 84b351425b4a43d68dacc18313dad243 P 828bb2f146144ed2826a65906ec39af5: Bootstrap starting.
I20260621 05:13:03.396337 18449 rpc_server.cc:307] RPC server started. Bound to: 127.3.131.129:45257
I20260621 05:13:03.396484 18607 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.131.129:45257 every 8 connection(s)
I20260621 05:13:03.396795 18449 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782018616648578-3598-0/minicluster-data/ts-0/data/info.pb
I20260621 05:13:03.400830 18500 log.cc:824] T 84b351425b4a43d68dacc18313dad243 P 828bb2f146144ed2826a65906ec39af5: Log is configured to *not* fsync() on all Append() calls
I20260621 05:13:03.402921 3598 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWZOx2_/build/release/bin/kudu as pid 18449
I20260621 05:13:03.403275 3598 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWZOx2_/build/release/bin/kudu
/tmp/dist-test-taskWZOx2_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782018616648578-3598-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782018616648578-3598-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782018616648578-3598-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782018616648578-3598-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.131.130:42965
--local_ip_for_outbound_sockets=127.3.131.130
--tserver_master_addrs=127.3.131.190:45579
--webserver_port=46151
--webserver_interface=127.3.131.130
--keytab_file=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782018616648578-3598-0/krb5kdc/kudu_127.3.131.130.keytab
--principal=kudu/127.3.131.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.131.148:41803
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782018616648578-3598-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782018616648578-3598-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782018616648578-3598-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260621 05:13:03.403821 18500 tablet_bootstrap.cc:492] T 84b351425b4a43d68dacc18313dad243 P 828bb2f146144ed2826a65906ec39af5: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260621 05:13:03.404150 18500 tablet_bootstrap.cc:492] T 84b351425b4a43d68dacc18313dad243 P 828bb2f146144ed2826a65906ec39af5: Bootstrap complete.
I20260621 05:13:03.405324 18500 ts_tablet_manager.cc:1397] T 84b351425b4a43d68dacc18313dad243 P 828bb2f146144ed2826a65906ec39af5: Time spent bootstrapping tablet: real 0.017s user 0.002s sys 0.012s
I20260621 05:13:03.406801 18500 raft_consensus.cc:348] T 84b351425b4a43d68dacc18313dad243 P 828bb2f146144ed2826a65906ec39af5 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0db2c38d6e21472abec748a4d51ff08b" member_type: VOTER last_known_addr { host: "127.3.131.131" port: 33449 } } peers { permanent_uuid: "090c4546eb4d4b02b3daa5bb23ec3444" member_type: VOTER last_known_addr { host: "127.3.131.130" port: 42965 } } peers { permanent_uuid: "828bb2f146144ed2826a65906ec39af5" member_type: VOTER last_known_addr { host: "127.3.131.129" port: 45257 } }
I20260621 05:13:03.407126 18500 raft_consensus.cc:729] T 84b351425b4a43d68dacc18313dad243 P 828bb2f146144ed2826a65906ec39af5 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 828bb2f146144ed2826a65906ec39af5, State: Initialized, Role: FOLLOWER
I20260621 05:13:03.407341 18500 consensus_queue.cc:260] T 84b351425b4a43d68dacc18313dad243 P 828bb2f146144ed2826a65906ec39af5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0db2c38d6e21472abec748a4d51ff08b" member_type: VOTER last_known_addr { host: "127.3.131.131" port: 33449 } } peers { permanent_uuid: "090c4546eb4d4b02b3daa5bb23ec3444" member_type: VOTER last_known_addr { host: "127.3.131.130" port: 42965 } } peers { permanent_uuid: "828bb2f146144ed2826a65906ec39af5" member_type: VOTER last_known_addr { host: "127.3.131.129" port: 45257 } }
I20260621 05:13:03.407644 18500 ts_tablet_manager.cc:1428] T 84b351425b4a43d68dacc18313dad243 P 828bb2f146144ed2826a65906ec39af5: Time spent starting tablet: real 0.002s user 0.004s sys 0.000s
W20260621 05:13:03.427544 18610 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 05:13:03.397622 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.131.190:45579 (local address 127.3.131.129:37063)
0621 05:13:03.397869 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0621 05:13:03.397921 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0621 05:13:03.398401 (+ 480us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:13:03.427328 (+ 28927us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.131.190:45579: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":193,"thread_start_us":110,"threads_started":1}
W20260621 05:13:03.427796 18608 heartbeater.cc:644] Failed to heartbeat to 127.3.131.190:45579 (0 consecutive failures): Network error: Failed to ping master at 127.3.131.190:45579: Client connection negotiation failed: client connection to 127.3.131.190:45579: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260621 05:13:03.428140 18610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:13:03.428033 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.131.190:45579 (local address 127.3.131.129:59577)
0621 05:13:03.428083 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0621 05:13:03.428122 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.131.190:45579: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
I20260621 05:13:03.428489 18610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:13:03.428384 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.131.190:45579 (local address 127.3.131.129:37983)
0621 05:13:03.428438 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0621 05:13:03.428462 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.131.190:45579: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260621 05:13:03.428602 18608 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260621 05:13:03.489918 18612 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 05:13:03.490181 18612 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 05:13:03.490226 18612 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 05:13:03.491590 18612 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 05:13:03.491654 18612 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 05:13:03.491712 18612 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.131.130
I20260621 05:13:03.493307 18612 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.131.148:41803
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782018616648578-3598-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782018616648578-3598-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782018616648578-3598-0/krb5kdc/kudu_127.3.131.130.keytab
--principal=kudu/127.3.131.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.131.130:42965
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782018616648578-3598-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.131.130
--webserver_port=46151
--tserver_master_addrs=127.3.131.190:45579
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18612
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.131.130
--log_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782018616648578-3598-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 21 Jun 2026 04:16:16 UTC on ccycloud.quasar-xycaha.root.comops.site
build id 4457881
I20260621 05:13:03.493606 18612 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 05:13:03.493870 18612 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 05:13:03.499413 18618 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:13:03.499419 18621 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:13:03.499437 18612 server_base.cc:1034] running on GCE node
W20260621 05:13:03.499419 18619 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:13:03.500028 18612 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 05:13:03.500336 18612 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 05:13:03.501531 18612 hybrid_clock.cc:648] HybridClock initialized: now 1782018783501521 us; error 32 us; skew 500 ppm
Jun 21 05:13:03 dist-test-slave-vrq1 krb5kdc[17306](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782018783, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.131.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 05:13:03.504596 18612 init.cc:377] Logged in from keytab as kudu/127.3.131.130@KRBTEST.COM (short username kudu)
I20260621 05:13:03.505085 18612 webserver.cc:460] Webserver started at http://127.3.131.130:46151/ using document root <none> and password file <none>
I20260621 05:13:03.505339 18612 fs_manager.cc:359] Metadata directory not provided
I20260621 05:13:03.505412 18612 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 05:13:03.506958 18612 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260621 05:13:03.507727 18628 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 05:13:03.507992 18612 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260621 05:13:03.508104 18612 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782018616648578-3598-0/minicluster-data/ts-1/data,/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782018616648578-3598-0/minicluster-data/ts-1/wal
uuid: "090c4546eb4d4b02b3daa5bb23ec3444"
format_stamp: "Formatted at 2026-06-21 05:13:01 on dist-test-slave-vrq1"
server_key: "87ccbc83f2293f21959f5ce191249c7d"
server_key_iv: "766838c1befa0d8cc7fb0ec6b5f15826"
server_key_version: "encryptionkey@0"
I20260621 05:13:03.508505 18612 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782018616648578-3598-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782018616648578-3598-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782018616648578-3598-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 05:13:03.522883 18612 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 05:13:03.525401 18612 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 05:13:03.525687 18612 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 05:13:03.525988 18612 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 05:13:03.526333 18612 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 05:13:03.526389 18612 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 05:13:03.526427 18612 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 05:13:03.526473 18612 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 05:13:03.532852 18612 rpc_server.cc:307] RPC server started. Bound to: 127.3.131.130:42965
I20260621 05:13:03.533000 18741 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.131.130:42965 every 8 connection(s)
I20260621 05:13:03.533329 18612 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782018616648578-3598-0/minicluster-data/ts-1/data/info.pb
I20260621 05:13:03.534746 18744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:13:03.534127 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.131.190:45579 (local address 127.3.131.130:57085)
0621 05:13:03.534305 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0621 05:13:03.534381 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.131.190:45579: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":71,"threads_started":1}
W20260621 05:13:03.535135 18742 heartbeater.cc:644] Failed to heartbeat to 127.3.131.190:45579 (0 consecutive failures): Network error: Failed to ping master at 127.3.131.190:45579: Client connection negotiation failed: client connection to 127.3.131.190:45579: connect: Connection refused (error 111)
I20260621 05:13:03.535555 18744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:13:03.535452 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.131.190:45579 (local address 127.3.131.130:60643)
0621 05:13:03.535502 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0621 05:13:03.535532 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.131.190:45579: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
I20260621 05:13:03.535964 18744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:13:03.535840 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.131.190:45579 (local address 127.3.131.130:36873)
0621 05:13:03.535879 (+ 39us) negotiation.cc:107] Waiting for socket to connect
0621 05:13:03.535947 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.131.190:45579: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
W20260621 05:13:03.536103 18742 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260621 05:13:03.540676 3598 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWZOx2_/build/release/bin/kudu as pid 18612
I20260621 05:13:03.541005 3598 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWZOx2_/build/release/bin/kudu
/tmp/dist-test-taskWZOx2_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782018616648578-3598-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782018616648578-3598-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782018616648578-3598-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782018616648578-3598-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.131.131:33449
--local_ip_for_outbound_sockets=127.3.131.131
--tserver_master_addrs=127.3.131.190:45579
--webserver_port=37631
--webserver_interface=127.3.131.131
--keytab_file=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782018616648578-3598-0/krb5kdc/kudu_127.3.131.131.keytab
--principal=kudu/127.3.131.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.131.148:41803
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782018616648578-3598-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782018616648578-3598-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782018616648578-3598-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 05:13:03.631500 18745 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 05:13:03.631750 18745 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 05:13:03.631784 18745 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 05:13:03.633172 18745 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 05:13:03.633234 18745 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 05:13:03.633267 18745 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.131.131
I20260621 05:13:03.634970 18745 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.131.148:41803
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782018616648578-3598-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782018616648578-3598-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782018616648578-3598-0/krb5kdc/kudu_127.3.131.131.keytab
--principal=kudu/127.3.131.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.131.131:33449
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782018616648578-3598-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.131.131
--webserver_port=37631
--tserver_master_addrs=127.3.131.190:45579
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18745
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.131.131
--log_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782018616648578-3598-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 21 Jun 2026 04:16:16 UTC on ccycloud.quasar-xycaha.root.comops.site
build id 4457881
I20260621 05:13:03.635322 18745 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 05:13:03.635617 18745 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 05:13:03.641216 18751 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:13:03.641214 18750 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:13:03.641212 18753 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:13:03.641713 18745 server_base.cc:1034] running on GCE node
I20260621 05:13:03.641933 18745 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 05:13:03.642165 18745 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 05:13:03.643373 18745 hybrid_clock.cc:648] HybridClock initialized: now 1782018783643356 us; error 40 us; skew 500 ppm
Jun 21 05:13:03 dist-test-slave-vrq1 krb5kdc[17306](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782018783, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.131.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 05:13:03.646595 18745 init.cc:377] Logged in from keytab as kudu/127.3.131.131@KRBTEST.COM (short username kudu)
I20260621 05:13:03.647105 18745 webserver.cc:460] Webserver started at http://127.3.131.131:37631/ using document root <none> and password file <none>
I20260621 05:13:03.647378 18745 fs_manager.cc:359] Metadata directory not provided
I20260621 05:13:03.647464 18745 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 05:13:03.649078 18745 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260621 05:13:03.649909 18760 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 05:13:03.650166 18745 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260621 05:13:03.650282 18745 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782018616648578-3598-0/minicluster-data/ts-2/data,/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782018616648578-3598-0/minicluster-data/ts-2/wal
uuid: "0db2c38d6e21472abec748a4d51ff08b"
format_stamp: "Formatted at 2026-06-21 05:13:02 on dist-test-slave-vrq1"
server_key: "1b0bfccd06bec7c991568420c076d8f7"
server_key_iv: "395637a84e8143975246366fff991031"
server_key_version: "encryptionkey@0"
I20260621 05:13:03.650698 18745 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782018616648578-3598-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782018616648578-3598-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782018616648578-3598-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 05:13:03.659933 18745 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 05:13:03.662775 18745 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 05:13:03.663005 18745 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 05:13:03.663357 18745 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 05:13:03.663705 18745 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 05:13:03.663761 18745 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 05:13:03.663805 18745 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 05:13:03.663837 18745 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 05:13:03.670073 18745 rpc_server.cc:307] RPC server started. Bound to: 127.3.131.131:33449
I20260621 05:13:03.670148 18873 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.131.131:33449 every 8 connection(s)
I20260621 05:13:03.670529 18745 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782018616648578-3598-0/minicluster-data/ts-2/data/info.pb
I20260621 05:13:03.672001 18876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:13:03.671378 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.131.190:45579 (local address 127.3.131.131:38183)
0621 05:13:03.671530 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0621 05:13:03.671605 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.131.190:45579: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":66,"threads_started":1}
W20260621 05:13:03.672240 18874 heartbeater.cc:644] Failed to heartbeat to 127.3.131.190:45579 (0 consecutive failures): Network error: Failed to ping master at 127.3.131.190:45579: Client connection negotiation failed: client connection to 127.3.131.190:45579: connect: Connection refused (error 111)
I20260621 05:13:03.672562 18876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:13:03.672465 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.131.190:45579 (local address 127.3.131.131:48305)
0621 05:13:03.672512 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0621 05:13:03.672544 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.131.190:45579: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
I20260621 05:13:03.672847 18876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:13:03.672762 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.131.190:45579 (local address 127.3.131.131:36019)
0621 05:13:03.672803 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0621 05:13:03.672824 (+ 21us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.131.190:45579: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
W20260621 05:13:03.672945 18874 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260621 05:13:03.677124 3598 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWZOx2_/build/release/bin/kudu as pid 18745
I20260621 05:13:04.429253 18878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:13:04.428987 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.131.190:45579 (local address 127.3.131.129:36977)
0621 05:13:04.429137 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0621 05:13:04.429182 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.131.190:45579: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":60,"threads_started":1}
I20260621 05:13:04.536767 18879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:13:04.536497 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.131.190:45579 (local address 127.3.131.130:48271)
0621 05:13:04.536651 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0621 05:13:04.536699 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.131.190:45579: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":62,"threads_started":1}
I20260621 05:13:04.673735 18880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:13:04.673412 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.131.190:45579 (local address 127.3.131.131:57433)
0621 05:13:04.673625 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0621 05:13:04.673668 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.131.190:45579: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":194,"thread_start_us":90,"threads_started":1}
I20260621 05:13:04.720986 18881 raft_consensus.cc:482] T 84b351425b4a43d68dacc18313dad243 P 828bb2f146144ed2826a65906ec39af5 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260621 05:13:04.721182 18881 raft_consensus.cc:504] T 84b351425b4a43d68dacc18313dad243 P 828bb2f146144ed2826a65906ec39af5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0db2c38d6e21472abec748a4d51ff08b" member_type: VOTER last_known_addr { host: "127.3.131.131" port: 33449 } } peers { permanent_uuid: "090c4546eb4d4b02b3daa5bb23ec3444" member_type: VOTER last_known_addr { host: "127.3.131.130" port: 42965 } } peers { permanent_uuid: "828bb2f146144ed2826a65906ec39af5" member_type: VOTER last_known_addr { host: "127.3.131.129" port: 45257 } }
I20260621 05:13:04.721680 18881 leader_election.cc:288] T 84b351425b4a43d68dacc18313dad243 P 828bb2f146144ed2826a65906ec39af5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0db2c38d6e21472abec748a4d51ff08b (127.3.131.131:33449), 090c4546eb4d4b02b3daa5bb23ec3444 (127.3.131.130:42965)
Jun 21 05:13:04 dist-test-slave-vrq1 krb5kdc[17306](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782018783, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.131.129@KRBTEST.COM for kudu/127.3.131.130@KRBTEST.COM
Jun 21 05:13:04 dist-test-slave-vrq1 krb5kdc[17306](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782018783, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.131.129@KRBTEST.COM for kudu/127.3.131.131@KRBTEST.COM
I20260621 05:13:04.733177 18883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:13:04.722096 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.131.129:50759 (local address 127.3.131.130:42965)
0621 05:13:04.722268 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0621 05:13:04.722278 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0621 05:13:04.722305 (+ 27us) server_negotiation.cc:407] Connection header received
0621 05:13:04.722732 (+ 427us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:13:04.722735 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:13:04.722774 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:13:04.722935 (+ 161us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:13:04.723796 (+ 861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:13:04.724752 (+ 956us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:13:04.725844 (+ 1092us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:13:04.726308 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:13:04.726451 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:13:04.729011 (+ 2560us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:13:04.729016 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:13:04.729017 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:13:04.729031 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:13:04.731033 (+ 2002us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:13:04.731735 (+ 702us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:13:04.731738 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:13:04.731738 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:13:04.731831 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:13:04.732247 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:13:04.732248 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:13:04.732248 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:13:04.732516 (+ 268us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:13:04.732617 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0621 05:13:04.732899 (+ 282us) server_negotiation.cc:299] Negotiation successful
0621 05:13:04.733034 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":123,"thread_start_us":61,"threads_started":1}
I20260621 05:13:04.733155 18878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:13:04.721927 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.131.130:42965 (local address 127.3.131.129:50759)
0621 05:13:04.721994 (+ 67us) negotiation.cc:107] Waiting for socket to connect
0621 05:13:04.722007 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0621 05:13:04.722068 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:13:04.722963 (+ 895us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:13:04.722966 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:13:04.723412 (+ 446us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:13:04.723618 (+ 206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:13:04.723625 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:13:04.724877 (+ 1252us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:13:04.724877 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:13:04.725673 (+ 796us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:13:04.725676 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:13:04.726399 (+ 723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:13:04.726400 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:13:04.726456 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:13:04.727052 (+ 596us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 05:13:04.727061 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:13:04.728831 (+ 1770us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 05:13:04.731157 (+ 2326us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:13:04.731166 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:13:04.731168 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:13:04.731572 (+ 404us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:13:04.731948 (+ 376us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:13:04.731949 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:13:04.731950 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:13:04.732144 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:13:04.732598 (+ 454us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 05:13:04.732599 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 05:13:04.732712 (+ 113us) client_negotiation.cc:769] Sending connection context
0621 05:13:04.732821 (+ 109us) client_negotiation.cc:240] Negotiation successful
0621 05:13:04.733041 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":45,"mutex_wait_us":93}
I20260621 05:13:04.733285 18882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:13:04.721973 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.131.131:33449 (local address 127.3.131.129:45809)
0621 05:13:04.722097 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0621 05:13:04.722106 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0621 05:13:04.722146 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:13:04.722963 (+ 817us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:13:04.722966 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:13:04.723454 (+ 488us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:13:04.723620 (+ 166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:13:04.723626 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:13:04.724866 (+ 1240us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:13:04.724867 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:13:04.725672 (+ 805us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:13:04.725674 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:13:04.726300 (+ 626us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:13:04.726301 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:13:04.726438 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:13:04.727102 (+ 664us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 05:13:04.727107 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:13:04.729158 (+ 2051us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 05:13:04.731390 (+ 2232us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:13:04.731392 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:13:04.731392 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:13:04.731599 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:13:04.731939 (+ 340us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:13:04.731940 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:13:04.731941 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:13:04.732098 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:13:04.732560 (+ 462us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 05:13:04.732567 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 05:13:04.732713 (+ 146us) client_negotiation.cc:769] Sending connection context
0621 05:13:04.732825 (+ 112us) client_negotiation.cc:240] Negotiation successful
0621 05:13:04.733132 (+ 307us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":112,"mutex_wait_us":147,"thread_start_us":69,"threads_started":1}
I20260621 05:13:04.733201 18884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:13:04.722097 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.131.129:45809 (local address 127.3.131.131:33449)
0621 05:13:04.722268 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0621 05:13:04.722278 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0621 05:13:04.722330 (+ 52us) server_negotiation.cc:407] Connection header received
0621 05:13:04.722737 (+ 407us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:13:04.722739 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:13:04.722774 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:13:04.722915 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:13:04.723776 (+ 861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:13:04.724745 (+ 969us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:13:04.725766 (+ 1021us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:13:04.726191 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:13:04.726332 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:13:04.729283 (+ 2951us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:13:04.729288 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:13:04.729288 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:13:04.729325 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:13:04.731306 (+ 1981us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:13:04.731688 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:13:04.731691 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:13:04.731692 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:13:04.731799 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:13:04.732193 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:13:04.732194 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:13:04.732194 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:13:04.732480 (+ 286us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:13:04.732589 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0621 05:13:04.732910 (+ 321us) server_negotiation.cc:299] Negotiation successful
0621 05:13:04.733071 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":134,"thread_start_us":58,"threads_started":1}
W20260621 05:13:04.734092 18495 leader_election.cc:341] T 84b351425b4a43d68dacc18313dad243 P 828bb2f146144ed2826a65906ec39af5 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 090c4546eb4d4b02b3daa5bb23ec3444 (127.3.131.130:42965): Not found: Tablet not found: 84b351425b4a43d68dacc18313dad243
W20260621 05:13:04.734205 18497 leader_election.cc:341] T 84b351425b4a43d68dacc18313dad243 P 828bb2f146144ed2826a65906ec39af5 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 0db2c38d6e21472abec748a4d51ff08b (127.3.131.131:33449): Not found: Tablet not found: 84b351425b4a43d68dacc18313dad243
I20260621 05:13:04.734254 18497 leader_election.cc:302] T 84b351425b4a43d68dacc18313dad243 P 828bb2f146144ed2826a65906ec39af5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 828bb2f146144ed2826a65906ec39af5; no voters: 090c4546eb4d4b02b3daa5bb23ec3444, 0db2c38d6e21472abec748a4d51ff08b
I20260621 05:13:04.734478 18881 raft_consensus.cc:2726] T 84b351425b4a43d68dacc18313dad243 P 828bb2f146144ed2826a65906ec39af5 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260621 05:13:05.429935 18885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:13:05.429725 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.131.190:45579 (local address 127.3.131.129:44039)
0621 05:13:05.429856 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0621 05:13:05.429889 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.131.190:45579: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":69,"threads_started":1}
I20260621 05:13:05.537578 18886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:13:05.537346 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.131.190:45579 (local address 127.3.131.130:40677)
0621 05:13:05.537492 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0621 05:13:05.537534 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.131.190:45579: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":62,"threads_started":1}
I20260621 05:13:05.674460 18887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:13:05.674267 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.131.190:45579 (local address 127.3.131.131:54509)
0621 05:13:05.674384 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0621 05:13:05.674419 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.131.190:45579: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":61,"threads_started":1}
I20260621 05:13:06.430787 18888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:13:06.430539 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.131.190:45579 (local address 127.3.131.129:49051)
0621 05:13:06.430689 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0621 05:13:06.430735 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.131.190:45579: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":63,"threads_started":1}
I20260621 05:13:06.538486 18889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:13:06.538195 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.131.190:45579 (local address 127.3.131.130:44963)
0621 05:13:06.538362 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0621 05:13:06.538436 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.131.190:45579: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":72,"threads_started":1}
I20260621 05:13:06.567886 18890 raft_consensus.cc:482] T 84b351425b4a43d68dacc18313dad243 P 828bb2f146144ed2826a65906ec39af5 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260621 05:13:06.568107 18890 raft_consensus.cc:504] T 84b351425b4a43d68dacc18313dad243 P 828bb2f146144ed2826a65906ec39af5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0db2c38d6e21472abec748a4d51ff08b" member_type: VOTER last_known_addr { host: "127.3.131.131" port: 33449 } } peers { permanent_uuid: "090c4546eb4d4b02b3daa5bb23ec3444" member_type: VOTER last_known_addr { host: "127.3.131.130" port: 42965 } } peers { permanent_uuid: "828bb2f146144ed2826a65906ec39af5" member_type: VOTER last_known_addr { host: "127.3.131.129" port: 45257 } }
I20260621 05:13:06.568346 18890 leader_election.cc:288] T 84b351425b4a43d68dacc18313dad243 P 828bb2f146144ed2826a65906ec39af5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0db2c38d6e21472abec748a4d51ff08b (127.3.131.131:33449), 090c4546eb4d4b02b3daa5bb23ec3444 (127.3.131.130:42965)
W20260621 05:13:06.569108 18497 leader_election.cc:341] T 84b351425b4a43d68dacc18313dad243 P 828bb2f146144ed2826a65906ec39af5 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 0db2c38d6e21472abec748a4d51ff08b (127.3.131.131:33449): Not found: Tablet not found: 84b351425b4a43d68dacc18313dad243
W20260621 05:13:06.569200 18495 leader_election.cc:341] T 84b351425b4a43d68dacc18313dad243 P 828bb2f146144ed2826a65906ec39af5 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 090c4546eb4d4b02b3daa5bb23ec3444 (127.3.131.130:42965): Not found: Tablet not found: 84b351425b4a43d68dacc18313dad243
I20260621 05:13:06.569231 18497 leader_election.cc:302] T 84b351425b4a43d68dacc18313dad243 P 828bb2f146144ed2826a65906ec39af5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 828bb2f146144ed2826a65906ec39af5; no voters: 090c4546eb4d4b02b3daa5bb23ec3444, 0db2c38d6e21472abec748a4d51ff08b
I20260621 05:13:06.569306 18890 raft_consensus.cc:2726] T 84b351425b4a43d68dacc18313dad243 P 828bb2f146144ed2826a65906ec39af5 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260621 05:13:06.675377 18891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:13:06.675027 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.131.190:45579 (local address 127.3.131.131:57011)
0621 05:13:06.675281 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0621 05:13:06.675330 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.131.190:45579: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":226,"thread_start_us":69,"threads_started":1}
I20260621 05:13:07.431667 18892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:13:07.431447 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.131.190:45579 (local address 127.3.131.129:32943)
0621 05:13:07.431575 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0621 05:13:07.431620 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.131.190:45579: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":61,"threads_started":1}
I20260621 05:13:07.539687 18893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:13:07.539440 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.131.190:45579 (local address 127.3.131.130:34051)
0621 05:13:07.539593 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0621 05:13:07.539638 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.131.190:45579: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":81,"threads_started":1}
I20260621 05:13:07.676100 18894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:13:07.675894 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.131.190:45579 (local address 127.3.131.131:38511)
0621 05:13:07.676022 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0621 05:13:07.676056 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.131.190:45579: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":72,"threads_started":1}
I20260621 05:13:08.432642 18895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:13:08.432372 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.131.190:45579 (local address 127.3.131.129:36471)
0621 05:13:08.432544 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0621 05:13:08.432596 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.131.190:45579: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":150,"thread_start_us":84,"threads_started":1}
I20260621 05:13:08.473017 18896 raft_consensus.cc:482] T 84b351425b4a43d68dacc18313dad243 P 828bb2f146144ed2826a65906ec39af5 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260621 05:13:08.473141 18896 raft_consensus.cc:504] T 84b351425b4a43d68dacc18313dad243 P 828bb2f146144ed2826a65906ec39af5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0db2c38d6e21472abec748a4d51ff08b" member_type: VOTER last_known_addr { host: "127.3.131.131" port: 33449 } } peers { permanent_uuid: "090c4546eb4d4b02b3daa5bb23ec3444" member_type: VOTER last_known_addr { host: "127.3.131.130" port: 42965 } } peers { permanent_uuid: "828bb2f146144ed2826a65906ec39af5" member_type: VOTER last_known_addr { host: "127.3.131.129" port: 45257 } }
I20260621 05:13:08.473412 18896 leader_election.cc:288] T 84b351425b4a43d68dacc18313dad243 P 828bb2f146144ed2826a65906ec39af5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0db2c38d6e21472abec748a4d51ff08b (127.3.131.131:33449), 090c4546eb4d4b02b3daa5bb23ec3444 (127.3.131.130:42965)
W20260621 05:13:08.473929 18497 leader_election.cc:341] T 84b351425b4a43d68dacc18313dad243 P 828bb2f146144ed2826a65906ec39af5 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 0db2c38d6e21472abec748a4d51ff08b (127.3.131.131:33449): Not found: Tablet not found: 84b351425b4a43d68dacc18313dad243
W20260621 05:13:08.474014 18495 leader_election.cc:341] T 84b351425b4a43d68dacc18313dad243 P 828bb2f146144ed2826a65906ec39af5 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 090c4546eb4d4b02b3daa5bb23ec3444 (127.3.131.130:42965): Not found: Tablet not found: 84b351425b4a43d68dacc18313dad243
I20260621 05:13:08.474079 18495 leader_election.cc:302] T 84b351425b4a43d68dacc18313dad243 P 828bb2f146144ed2826a65906ec39af5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 828bb2f146144ed2826a65906ec39af5; no voters: 090c4546eb4d4b02b3daa5bb23ec3444, 0db2c38d6e21472abec748a4d51ff08b
I20260621 05:13:08.474217 18896 raft_consensus.cc:2726] T 84b351425b4a43d68dacc18313dad243 P 828bb2f146144ed2826a65906ec39af5 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260621 05:13:08.540643 18897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:13:08.540376 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.131.190:45579 (local address 127.3.131.130:34149)
0621 05:13:08.540561 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0621 05:13:08.540595 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.131.190:45579: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":160,"thread_start_us":120,"threads_started":1}
I20260621 05:13:08.676996 18898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:13:08.676741 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.131.190:45579 (local address 127.3.131.131:58995)
0621 05:13:08.676896 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0621 05:13:08.676945 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.131.190:45579: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":83,"threads_started":1}
I20260621 05:13:09.433636 18899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:13:09.433347 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.131.190:45579 (local address 127.3.131.129:35071)
0621 05:13:09.433541 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0621 05:13:09.433584 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.131.190:45579: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":164,"thread_start_us":121,"threads_started":1}
I20260621 05:13:09.541543 18900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:13:09.541276 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.131.190:45579 (local address 127.3.131.130:49487)
0621 05:13:09.541440 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0621 05:13:09.541489 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.131.190:45579: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":68,"threads_started":1}
I20260621 05:13:09.677945 18901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:13:09.677632 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.131.190:45579 (local address 127.3.131.131:35051)
0621 05:13:09.677800 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0621 05:13:09.677895 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.131.190:45579: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":81,"threads_started":1}
I20260621 05:13:10.434595 18902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:13:10.434296 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.131.190:45579 (local address 127.3.131.129:39667)
0621 05:13:10.434483 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0621 05:13:10.434534 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.131.190:45579: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":164,"thread_start_us":76,"threads_started":1}
I20260621 05:13:10.542496 18903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:13:10.542224 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.131.190:45579 (local address 127.3.131.130:54881)
0621 05:13:10.542401 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0621 05:13:10.542447 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.131.190:45579: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":155,"thread_start_us":79,"threads_started":1}
I20260621 05:13:10.678889 18904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:13:10.678627 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.131.190:45579 (local address 127.3.131.131:56955)
0621 05:13:10.678797 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0621 05:13:10.678843 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.131.190:45579: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":76,"threads_started":1}
I20260621 05:13:11.435556 18905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:13:11.435261 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.131.190:45579 (local address 127.3.131.129:34143)
0621 05:13:11.435458 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0621 05:13:11.435508 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.131.190:45579: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":173,"thread_start_us":74,"threads_started":1}
I20260621 05:13:11.543315 18906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:13:11.543020 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.131.190:45579 (local address 127.3.131.130:54809)
0621 05:13:11.543158 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0621 05:13:11.543267 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.131.190:45579: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":62,"threads_started":1}
I20260621 05:13:11.679801 18907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:13:11.679534 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.131.190:45579 (local address 127.3.131.131:49707)
0621 05:13:11.679702 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0621 05:13:11.679752 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.131.190:45579: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":71,"threads_started":1}
I20260621 05:13:11.980917 18908 raft_consensus.cc:482] T 84b351425b4a43d68dacc18313dad243 P 828bb2f146144ed2826a65906ec39af5 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260621 05:13:11.981025 18908 raft_consensus.cc:504] T 84b351425b4a43d68dacc18313dad243 P 828bb2f146144ed2826a65906ec39af5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0db2c38d6e21472abec748a4d51ff08b" member_type: VOTER last_known_addr { host: "127.3.131.131" port: 33449 } } peers { permanent_uuid: "090c4546eb4d4b02b3daa5bb23ec3444" member_type: VOTER last_known_addr { host: "127.3.131.130" port: 42965 } } peers { permanent_uuid: "828bb2f146144ed2826a65906ec39af5" member_type: VOTER last_known_addr { host: "127.3.131.129" port: 45257 } }
I20260621 05:13:11.981389 18908 leader_election.cc:288] T 84b351425b4a43d68dacc18313dad243 P 828bb2f146144ed2826a65906ec39af5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0db2c38d6e21472abec748a4d51ff08b (127.3.131.131:33449), 090c4546eb4d4b02b3daa5bb23ec3444 (127.3.131.130:42965)
W20260621 05:13:11.981626 18495 leader_election.cc:341] T 84b351425b4a43d68dacc18313dad243 P 828bb2f146144ed2826a65906ec39af5 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 090c4546eb4d4b02b3daa5bb23ec3444 (127.3.131.130:42965): Not found: Tablet not found: 84b351425b4a43d68dacc18313dad243
W20260621 05:13:11.981691 18497 leader_election.cc:341] T 84b351425b4a43d68dacc18313dad243 P 828bb2f146144ed2826a65906ec39af5 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 0db2c38d6e21472abec748a4d51ff08b (127.3.131.131:33449): Not found: Tablet not found: 84b351425b4a43d68dacc18313dad243
I20260621 05:13:11.981726 18497 leader_election.cc:302] T 84b351425b4a43d68dacc18313dad243 P 828bb2f146144ed2826a65906ec39af5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 828bb2f146144ed2826a65906ec39af5; no voters: 090c4546eb4d4b02b3daa5bb23ec3444, 0db2c38d6e21472abec748a4d51ff08b
I20260621 05:13:11.981797 18908 raft_consensus.cc:2726] T 84b351425b4a43d68dacc18313dad243 P 828bb2f146144ed2826a65906ec39af5 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260621 05:13:12.436414 18909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:13:12.436159 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.131.190:45579 (local address 127.3.131.129:41927)
0621 05:13:12.436319 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0621 05:13:12.436366 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.131.190:45579: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":68,"threads_started":1}
I20260621 05:13:12.544140 18910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:13:12.543896 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.131.190:45579 (local address 127.3.131.130:36985)
0621 05:13:12.544047 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0621 05:13:12.544094 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.131.190:45579: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":81,"threads_started":1}
I20260621 05:13:12.680596 18911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:13:12.680370 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.131.190:45579 (local address 127.3.131.131:41509)
0621 05:13:12.680513 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0621 05:13:12.680551 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.131.190:45579: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":63,"threads_started":1}
I20260621 05:13:13.437299 18912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:13:13.437036 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.131.190:45579 (local address 127.3.131.129:51291)
0621 05:13:13.437206 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0621 05:13:13.437252 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.131.190:45579: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":149,"thread_start_us":73,"threads_started":1}
I20260621 05:13:13.545094 18913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:13:13.544709 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.131.190:45579 (local address 127.3.131.130:55309)
0621 05:13:13.544907 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0621 05:13:13.545003 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.131.190:45579: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":172,"thread_start_us":123,"threads_started":1}
I20260621 05:13:13.681305 18914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:13:13.681097 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.131.190:45579 (local address 127.3.131.131:52939)
0621 05:13:13.681220 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0621 05:13:13.681258 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.131.190:45579: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":60,"threads_started":1}
I20260621 05:13:14.438175 18915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:13:14.437878 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.131.190:45579 (local address 127.3.131.129:35263)
0621 05:13:14.438088 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0621 05:13:14.438130 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.131.190:45579: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":97,"threads_started":1}
I20260621 05:13:14.545990 18916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:13:14.545692 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.131.190:45579 (local address 127.3.131.130:36845)
0621 05:13:14.545898 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0621 05:13:14.545942 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.131.190:45579: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":154,"thread_start_us":85,"threads_started":1}
I20260621 05:13:14.682072 18917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:13:14.681817 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.131.190:45579 (local address 127.3.131.131:37035)
0621 05:13:14.681986 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0621 05:13:14.682028 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.131.190:45579: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":149,"thread_start_us":80,"threads_started":1}
I20260621 05:13:15.439092 18918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:13:15.438823 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.131.190:45579 (local address 127.3.131.129:59039)
0621 05:13:15.438998 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0621 05:13:15.439042 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.131.190:45579: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":151,"thread_start_us":82,"threads_started":1}
I20260621 05:13:15.546720 18919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:13:15.546497 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.131.190:45579 (local address 127.3.131.130:56791)
0621 05:13:15.546631 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0621 05:13:15.546673 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.131.190:45579: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":59,"threads_started":1}
I20260621 05:13:15.590106 18920 raft_consensus.cc:482] T 84b351425b4a43d68dacc18313dad243 P 828bb2f146144ed2826a65906ec39af5 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260621 05:13:15.590227 18920 raft_consensus.cc:504] T 84b351425b4a43d68dacc18313dad243 P 828bb2f146144ed2826a65906ec39af5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0db2c38d6e21472abec748a4d51ff08b" member_type: VOTER last_known_addr { host: "127.3.131.131" port: 33449 } } peers { permanent_uuid: "090c4546eb4d4b02b3daa5bb23ec3444" member_type: VOTER last_known_addr { host: "127.3.131.130" port: 42965 } } peers { permanent_uuid: "828bb2f146144ed2826a65906ec39af5" member_type: VOTER last_known_addr { host: "127.3.131.129" port: 45257 } }
I20260621 05:13:15.590467 18920 leader_election.cc:288] T 84b351425b4a43d68dacc18313dad243 P 828bb2f146144ed2826a65906ec39af5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0db2c38d6e21472abec748a4d51ff08b (127.3.131.131:33449), 090c4546eb4d4b02b3daa5bb23ec3444 (127.3.131.130:42965)
W20260621 05:13:15.590883 18497 leader_election.cc:341] T 84b351425b4a43d68dacc18313dad243 P 828bb2f146144ed2826a65906ec39af5 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 0db2c38d6e21472abec748a4d51ff08b (127.3.131.131:33449): Not found: Tablet not found: 84b351425b4a43d68dacc18313dad243
W20260621 05:13:15.590957 18495 leader_election.cc:341] T 84b351425b4a43d68dacc18313dad243 P 828bb2f146144ed2826a65906ec39af5 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 090c4546eb4d4b02b3daa5bb23ec3444 (127.3.131.130:42965): Not found: Tablet not found: 84b351425b4a43d68dacc18313dad243
I20260621 05:13:15.591055 18495 leader_election.cc:302] T 84b351425b4a43d68dacc18313dad243 P 828bb2f146144ed2826a65906ec39af5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 828bb2f146144ed2826a65906ec39af5; no voters: 090c4546eb4d4b02b3daa5bb23ec3444, 0db2c38d6e21472abec748a4d51ff08b
I20260621 05:13:15.591150 18920 raft_consensus.cc:2726] T 84b351425b4a43d68dacc18313dad243 P 828bb2f146144ed2826a65906ec39af5 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260621 05:13:15.682866 18921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:13:15.682620 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.131.190:45579 (local address 127.3.131.131:49739)
0621 05:13:15.682778 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0621 05:13:15.682819 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.131.190:45579: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":64,"threads_started":1}
I20260621 05:13:16.439958 18922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:13:16.439725 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.131.190:45579 (local address 127.3.131.129:50837)
0621 05:13:16.439878 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0621 05:13:16.439914 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.131.190:45579: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":61,"threads_started":1}
I20260621 05:13:16.547555 18923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:13:16.547337 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.131.190:45579 (local address 127.3.131.130:50937)
0621 05:13:16.547464 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0621 05:13:16.547506 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.131.190:45579: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":60,"threads_started":1}
I20260621 05:13:16.683918 18924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:13:16.683557 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.131.190:45579 (local address 127.3.131.131:44781)
0621 05:13:16.683797 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0621 05:13:16.683845 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.131.190:45579: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":210,"thread_start_us":99,"threads_started":1}
I20260621 05:13:17.440747 18925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:13:17.440512 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.131.190:45579 (local address 127.3.131.129:41423)
0621 05:13:17.440654 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0621 05:13:17.440700 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.131.190:45579: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":96,"threads_started":1}
I20260621 05:13:17.548544 18926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:13:17.548205 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.131.190:45579 (local address 127.3.131.130:39093)
0621 05:13:17.548420 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0621 05:13:17.548481 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.131.190:45579: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":198,"thread_start_us":99,"threads_started":1}
I20260621 05:13:17.684734 18927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:13:17.684448 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.131.190:45579 (local address 127.3.131.131:40293)
0621 05:13:17.684624 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0621 05:13:17.684669 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.131.190:45579: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":155,"thread_start_us":81,"threads_started":1}
I20260621 05:13:18.441625 18928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:13:18.441351 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.131.190:45579 (local address 127.3.131.129:60799)
0621 05:13:18.441512 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0621 05:13:18.441564 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.131.190:45579: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":64,"threads_started":1}
I20260621 05:13:18.549438 18929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:13:18.549171 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.131.190:45579 (local address 127.3.131.130:56791)
0621 05:13:18.549336 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0621 05:13:18.549386 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.131.190:45579: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":66,"threads_started":1}
I20260621 05:13:18.685716 18930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:13:18.685381 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.131.190:45579 (local address 127.3.131.131:42069)
0621 05:13:18.685570 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0621 05:13:18.685624 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.131.190:45579: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":165,"thread_start_us":71,"threads_started":1}
I20260621 05:13:18.686700 3598 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWZOx2_/build/release/bin/kudu with pid 18449
I20260621 05:13:18.691979 3598 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWZOx2_/build/release/bin/kudu with pid 18612
I20260621 05:13:18.696871 3598 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWZOx2_/build/release/bin/kudu with pid 18745
2026-06-21T05:13:18Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (18613 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782018616648578-3598-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 21 05:13:18 dist-test-slave-vrq1 krb5kdc[18933](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 21 05:13:18 dist-test-slave-vrq1 krb5kdc[18933](info): set up 2 sockets
Jun 21 05:13:18 dist-test-slave-vrq1 krb5kdc[18933](info): commencing operation
krb5kdc: starting...
W20260621 05:13:19.787539 3598 mini_kdc.cc:121] Time spent starting KDC: real 1.067s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782018616648578-3598-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782018616648578-3598-0/krb5kdc/test-user.keytab.
Jun 21 05:13:19 dist-test-slave-vrq1 krb5kdc[18933](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782018799, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-21T05:13:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-21T05:13:19Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.131.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.131.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.131.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782018616648578-3598-0/krb5kdc/kudu_127.3.131.190.keytab.
Entry for principal kudu/127.3.131.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782018616648578-3598-0/krb5kdc/kudu_127.3.131.190.keytab.
I20260621 05:13:19.918298 3598 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWZOx2_/build/release/bin/kudu
/tmp/dist-test-taskWZOx2_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782018616648578-3598-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782018616648578-3598-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782018616648578-3598-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782018616648578-3598-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.131.190:46697
--webserver_interface=127.3.131.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782018616648578-3598-0/krb5kdc/kudu_127.3.131.190.keytab
--principal=kudu/127.3.131.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.131.148:44007
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.131.190:46697
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782018616648578-3598-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782018616648578-3598-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782018616648578-3598-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 05:13:20.001839 18946 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 05:13:20.002082 18946 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 05:13:20.002108 18946 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 05:13:20.003391 18946 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260621 05:13:20.003438 18946 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260621 05:13:20.003453 18946 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260621 05:13:20.003464 18946 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 05:13:20.003479 18946 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 05:13:20.003492 18946 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 05:13:20.003504 18946 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 05:13:20.003517 18946 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 05:13:20.005087 18946 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.131.148:44007
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782018616648578-3598-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782018616648578-3598-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.3.131.190:46697
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782018616648578-3598-0/krb5kdc/kudu_127.3.131.190.keytab
--principal=kudu/127.3.131.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.131.190:46697
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782018616648578-3598-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.131.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18946
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782018616648578-3598-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 21 Jun 2026 04:16:16 UTC on ccycloud.quasar-xycaha.root.comops.site
build id 4457881
I20260621 05:13:20.005308 18946 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 05:13:20.005491 18946 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 05:13:20.012068 18951 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:13:20.012068 18954 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:13:20.012327 18946 server_base.cc:1034] running on GCE node
W20260621 05:13:20.012282 18952 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:13:20.012724 18946 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 05:13:20.013057 18946 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 05:13:20.014231 18946 hybrid_clock.cc:648] HybridClock initialized: now 1782018800014204 us; error 40 us; skew 500 ppm
Jun 21 05:13:20 dist-test-slave-vrq1 krb5kdc[18933](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782018800, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.131.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 05:13:20.017001 18946 init.cc:377] Logged in from keytab as kudu/127.3.131.190@KRBTEST.COM (short username kudu)
I20260621 05:13:20.017529 18946 webserver.cc:460] Webserver started at http://127.3.131.190:35139/ using document root <none> and password file <none>
I20260621 05:13:20.017835 18946 fs_manager.cc:359] Metadata directory not provided
I20260621 05:13:20.017887 18946 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 05:13:20.017984 18946 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 05:13:20.019047 18946 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782018616648578-3598-0/minicluster-data/master-0/data/instance:
uuid: "5b3269e323af422bb256795282d54866"
format_stamp: "Formatted at 2026-06-21 05:13:20 on dist-test-slave-vrq1"
server_key: "456833c8468b3a016c85b28e4daff374"
server_key_iv: "0b2b5d432553b57c179b0a4863527276"
server_key_version: "encryptionkey@0"
I20260621 05:13:20.019433 18946 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782018616648578-3598-0/minicluster-data/master-0/wal/instance:
uuid: "5b3269e323af422bb256795282d54866"
format_stamp: "Formatted at 2026-06-21 05:13:20 on dist-test-slave-vrq1"
server_key: "456833c8468b3a016c85b28e4daff374"
server_key_iv: "0b2b5d432553b57c179b0a4863527276"
server_key_version: "encryptionkey@0"
I20260621 05:13:20.021340 18946 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.002s sys 0.000s
I20260621 05:13:20.022476 18961 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 05:13:20.022734 18946 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260621 05:13:20.022840 18946 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782018616648578-3598-0/minicluster-data/master-0/data,/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782018616648578-3598-0/minicluster-data/master-0/wal
uuid: "5b3269e323af422bb256795282d54866"
format_stamp: "Formatted at 2026-06-21 05:13:20 on dist-test-slave-vrq1"
server_key: "456833c8468b3a016c85b28e4daff374"
server_key_iv: "0b2b5d432553b57c179b0a4863527276"
server_key_version: "encryptionkey@0"
I20260621 05:13:20.022945 18946 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782018616648578-3598-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782018616648578-3598-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782018616648578-3598-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 05:13:20.043099 18946 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 05:13:20.045682 18946 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 05:13:20.045814 18946 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 05:13:20.050006 18946 rpc_server.cc:307] RPC server started. Bound to: 127.3.131.190:46697
I20260621 05:13:20.050040 19013 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.131.190:46697 every 8 connection(s)
I20260621 05:13:20.050420 18946 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782018616648578-3598-0/minicluster-data/master-0/data/info.pb
I20260621 05:13:20.051499 19014 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 05:13:20.053817 19014 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5b3269e323af422bb256795282d54866: Bootstrap starting.
I20260621 05:13:20.054497 19014 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5b3269e323af422bb256795282d54866: Neither blocks nor log segments found. Creating new log.
I20260621 05:13:20.054483 3598 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWZOx2_/build/release/bin/kudu as pid 18946
I20260621 05:13:20.054590 3598 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782018616648578-3598-0/minicluster-data/master-0/wal/instance
I20260621 05:13:20.054816 3598 external_mini_cluster.cc:1428] Setting key 6f4219e26ca1102b46af98a46785d95e
I20260621 05:13:20.054859 19014 log.cc:824] T 00000000000000000000000000000000 P 5b3269e323af422bb256795282d54866: Log is configured to *not* fsync() on all Append() calls
I20260621 05:13:20.055768 19014 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5b3269e323af422bb256795282d54866: No bootstrap required, opened a new log
I20260621 05:13:20.058215 19014 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5b3269e323af422bb256795282d54866 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b3269e323af422bb256795282d54866" member_type: VOTER last_known_addr { host: "127.3.131.190" port: 46697 } }
I20260621 05:13:20.058434 19014 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5b3269e323af422bb256795282d54866 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 05:13:20.058490 19014 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5b3269e323af422bb256795282d54866 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b3269e323af422bb256795282d54866, State: Initialized, Role: FOLLOWER
I20260621 05:13:20.058668 19014 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5b3269e323af422bb256795282d54866 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b3269e323af422bb256795282d54866" member_type: VOTER last_known_addr { host: "127.3.131.190" port: 46697 } }
I20260621 05:13:20.058805 19014 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5b3269e323af422bb256795282d54866 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260621 05:13:20.058854 19014 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5b3269e323af422bb256795282d54866 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260621 05:13:20.058903 19014 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5b3269e323af422bb256795282d54866 [term 0 FOLLOWER]: Advancing to term 1
I20260621 05:13:20.059607 19014 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5b3269e323af422bb256795282d54866 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b3269e323af422bb256795282d54866" member_type: VOTER last_known_addr { host: "127.3.131.190" port: 46697 } }
I20260621 05:13:20.059739 19014 leader_election.cc:302] T 00000000000000000000000000000000 P 5b3269e323af422bb256795282d54866 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5b3269e323af422bb256795282d54866; no voters:
I20260621 05:13:20.059980 19014 leader_election.cc:288] T 00000000000000000000000000000000 P 5b3269e323af422bb256795282d54866 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260621 05:13:20.060014 19019 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5b3269e323af422bb256795282d54866 [term 1 FOLLOWER]: Leader election won for term 1
I20260621 05:13:20.060127 19019 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5b3269e323af422bb256795282d54866 [term 1 LEADER]: Becoming Leader. State: Replica: 5b3269e323af422bb256795282d54866, State: Running, Role: LEADER
I20260621 05:13:20.060279 19014 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5b3269e323af422bb256795282d54866 [sys.catalog]: configured and running, proceeding with master startup.
I20260621 05:13:20.060256 19019 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5b3269e323af422bb256795282d54866 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b3269e323af422bb256795282d54866" member_type: VOTER last_known_addr { host: "127.3.131.190" port: 46697 } }
I20260621 05:13:20.062011 19021 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5b3269e323af422bb256795282d54866 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5b3269e323af422bb256795282d54866. Latest consensus state: current_term: 1 leader_uuid: "5b3269e323af422bb256795282d54866" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b3269e323af422bb256795282d54866" member_type: VOTER last_known_addr { host: "127.3.131.190" port: 46697 } } }
I20260621 05:13:20.062124 19021 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5b3269e323af422bb256795282d54866 [sys.catalog]: This master's current role is: LEADER
W20260621 05:13:20.062235 19034 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 5b3269e323af422bb256795282d54866: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260621 05:13:20.062304 19034 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260621 05:13:20.062337 19020 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5b3269e323af422bb256795282d54866 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5b3269e323af422bb256795282d54866" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b3269e323af422bb256795282d54866" member_type: VOTER last_known_addr { host: "127.3.131.190" port: 46697 } } }
I20260621 05:13:20.062412 19020 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5b3269e323af422bb256795282d54866 [sys.catalog]: This master's current role is: LEADER
Jun 21 05:13:20 dist-test-slave-vrq1 krb5kdc[18933](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782018799, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.131.190@KRBTEST.COM
I20260621 05:13:20.062536 19036 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260621 05:13:20.062767 19036 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260621 05:13:20.065455 19036 catalog_manager.cc:1347] Generated new cluster ID: dce3d0e84ae94f97814f61bb245e0d57
I20260621 05:13:20.065532 19036 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260621 05:13:20.067090 19018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:13:20.055512 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39106 (local address 127.3.131.190:46697)
0621 05:13:20.055705 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0621 05:13:20.055715 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0621 05:13:20.055759 (+ 44us) server_negotiation.cc:407] Connection header received
0621 05:13:20.056279 (+ 520us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:13:20.056287 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:13:20.056343 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:13:20.056509 (+ 166us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:13:20.057266 (+ 757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:13:20.058193 (+ 927us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:13:20.059221 (+ 1028us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:13:20.059675 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:13:20.059808 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:13:20.063086 (+ 3278us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:13:20.063091 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:13:20.063091 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:13:20.063106 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:13:20.065331 (+ 2225us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:13:20.065776 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:13:20.065779 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:13:20.065780 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:13:20.065891 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:13:20.066125 (+ 234us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:13:20.066126 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:13:20.066126 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:13:20.066440 (+ 314us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:13:20.066592 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0621 05:13:20.066732 (+ 140us) server_negotiation.cc:299] Negotiation successful
0621 05:13:20.066873 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":133,"thread_start_us":88,"threads_started":1}
I20260621 05:13:20.089133 19036 catalog_manager.cc:1370] Generated new certificate authority record
I20260621 05:13:20.089789 19036 catalog_manager.cc:1504] Loading token signing keys...
I20260621 05:13:20.094321 19036 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5b3269e323af422bb256795282d54866: Generated new TSK 0
I20260621 05:13:20.094552 19036 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.3.131.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.131.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.131.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782018616648578-3598-0/krb5kdc/kudu_127.3.131.129.keytab.
Entry for principal kudu/127.3.131.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782018616648578-3598-0/krb5kdc/kudu_127.3.131.129.keytab.
I20260621 05:13:20.127454 3598 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWZOx2_/build/release/bin/kudu
/tmp/dist-test-taskWZOx2_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782018616648578-3598-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782018616648578-3598-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782018616648578-3598-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782018616648578-3598-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.131.129:0
--local_ip_for_outbound_sockets=127.3.131.129
--webserver_interface=127.3.131.129
--webserver_port=0
--tserver_master_addrs=127.3.131.190:46697
--keytab_file=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782018616648578-3598-0/krb5kdc/kudu_127.3.131.129.keytab
--principal=kudu/127.3.131.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.131.148:44007
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782018616648578-3598-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782018616648578-3598-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782018616648578-3598-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 05:13:20.211323 19040 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 05:13:20.211524 19040 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 05:13:20.211547 19040 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 05:13:20.212728 19040 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 05:13:20.212769 19040 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 05:13:20.212796 19040 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.131.129
I20260621 05:13:20.214242 19040 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.131.148:44007
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782018616648578-3598-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782018616648578-3598-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782018616648578-3598-0/krb5kdc/kudu_127.3.131.129.keytab
--principal=kudu/127.3.131.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.131.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782018616648578-3598-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.131.129
--webserver_port=0
--tserver_master_addrs=127.3.131.190:46697
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19040
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.131.129
--log_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782018616648578-3598-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 21 Jun 2026 04:16:16 UTC on ccycloud.quasar-xycaha.root.comops.site
build id 4457881
I20260621 05:13:20.214493 19040 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 05:13:20.214740 19040 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 05:13:20.220148 19046 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:13:20.220145 19048 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:13:20.220422 19040 server_base.cc:1034] running on GCE node
W20260621 05:13:20.220139 19045 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:13:20.220743 19040 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 05:13:20.221009 19040 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 05:13:20.222198 19040 hybrid_clock.cc:648] HybridClock initialized: now 1782018800222184 us; error 31 us; skew 500 ppm
Jun 21 05:13:20 dist-test-slave-vrq1 krb5kdc[18933](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782018800, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.131.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 05:13:20.224938 19040 init.cc:377] Logged in from keytab as kudu/127.3.131.129@KRBTEST.COM (short username kudu)
I20260621 05:13:20.225449 19040 webserver.cc:460] Webserver started at http://127.3.131.129:34091/ using document root <none> and password file <none>
I20260621 05:13:20.225689 19040 fs_manager.cc:359] Metadata directory not provided
I20260621 05:13:20.225759 19040 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 05:13:20.225914 19040 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 05:13:20.227067 19040 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782018616648578-3598-0/minicluster-data/ts-0/data/instance:
uuid: "af09019aeb194a60b128a035658ed606"
format_stamp: "Formatted at 2026-06-21 05:13:20 on dist-test-slave-vrq1"
server_key: "3a7917f757ceb859a74758824cce2404"
server_key_iv: "6d67965e121b33fd4eed38e9a363377a"
server_key_version: "encryptionkey@0"
I20260621 05:13:20.227517 19040 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782018616648578-3598-0/minicluster-data/ts-0/wal/instance:
uuid: "af09019aeb194a60b128a035658ed606"
format_stamp: "Formatted at 2026-06-21 05:13:20 on dist-test-slave-vrq1"
server_key: "3a7917f757ceb859a74758824cce2404"
server_key_iv: "6d67965e121b33fd4eed38e9a363377a"
server_key_version: "encryptionkey@0"
I20260621 05:13:20.228960 19040 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260621 05:13:20.229995 19055 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 05:13:20.230222 19040 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260621 05:13:20.230345 19040 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782018616648578-3598-0/minicluster-data/ts-0/data,/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782018616648578-3598-0/minicluster-data/ts-0/wal
uuid: "af09019aeb194a60b128a035658ed606"
format_stamp: "Formatted at 2026-06-21 05:13:20 on dist-test-slave-vrq1"
server_key: "3a7917f757ceb859a74758824cce2404"
server_key_iv: "6d67965e121b33fd4eed38e9a363377a"
server_key_version: "encryptionkey@0"
I20260621 05:13:20.230425 19040 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782018616648578-3598-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782018616648578-3598-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782018616648578-3598-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 05:13:20.247339 19040 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 05:13:20.249578 19040 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 05:13:20.249752 19040 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 05:13:20.250003 19040 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 05:13:20.250330 19040 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 05:13:20.250384 19040 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 05:13:20.250425 19040 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 05:13:20.250455 19040 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 05:13:20.256579 19040 rpc_server.cc:307] RPC server started. Bound to: 127.3.131.129:41035
I20260621 05:13:20.256626 19168 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.131.129:41035 every 8 connection(s)
I20260621 05:13:20.256985 19040 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782018616648578-3598-0/minicluster-data/ts-0/data/info.pb
I20260621 05:13:20.262933 3598 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWZOx2_/build/release/bin/kudu as pid 19040
I20260621 05:13:20.263062 3598 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782018616648578-3598-0/minicluster-data/ts-0/wal/instance
I20260621 05:13:20.263336 3598 external_mini_cluster.cc:1428] Setting key 10533ddd7de492738d6d72a866e40e2e
Jun 21 05:13:20 dist-test-slave-vrq1 krb5kdc[18933](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782018800, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.131.129@KRBTEST.COM for kudu/127.3.131.190@KRBTEST.COM
I20260621 05:13:20.268119 19018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:13:20.257799 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.131.129:39453 (local address 127.3.131.190:46697)
0621 05:13:20.257884 (+ 85us) server_negotiation.cc:206] Beginning negotiation
0621 05:13:20.257886 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:13:20.258073 (+ 187us) server_negotiation.cc:407] Connection header received
0621 05:13:20.258627 (+ 554us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:13:20.258629 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:13:20.258640 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:13:20.258701 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:13:20.259836 (+ 1135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:13:20.260488 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:13:20.261786 (+ 1298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:13:20.261996 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:13:20.262130 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:13:20.264891 (+ 2761us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:13:20.264893 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:13:20.264893 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:13:20.264903 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:13:20.266577 (+ 1674us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:13:20.267041 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:13:20.267042 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:13:20.267043 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:13:20.267104 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:13:20.267433 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:13:20.267434 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:13:20.267435 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:13:20.267557 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:13:20.267637 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0621 05:13:20.267963 (+ 326us) server_negotiation.cc:299] Negotiation successful
0621 05:13:20.268053 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260621 05:13:20.268618 19171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:13:20.257732 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.131.190:46697 (local address 127.3.131.129:39453)
0621 05:13:20.257963 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0621 05:13:20.258001 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0621 05:13:20.258511 (+ 510us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:13:20.258821 (+ 310us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:13:20.258825 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:13:20.259484 (+ 659us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:13:20.259713 (+ 229us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:13:20.259718 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:13:20.260601 (+ 883us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:13:20.260602 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:13:20.261666 (+ 1064us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:13:20.261668 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:13:20.262140 (+ 472us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:13:20.262141 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:13:20.262260 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:13:20.262917 (+ 657us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 05:13:20.262926 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:13:20.264766 (+ 1840us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 05:13:20.266689 (+ 1923us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:13:20.266697 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:13:20.266700 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:13:20.266947 (+ 247us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:13:20.267240 (+ 293us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:13:20.267241 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:13:20.267241 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:13:20.267357 (+ 116us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:13:20.267628 (+ 271us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 05:13:20.267635 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 05:13:20.267787 (+ 152us) client_negotiation.cc:769] Sending connection context
0621 05:13:20.268247 (+ 460us) client_negotiation.cc:240] Negotiation successful
0621 05:13:20.268437 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":192,"thread_start_us":140,"threads_started":1}
I20260621 05:13:20.269272 19169 heartbeater.cc:347] Connected to a master server at 127.3.131.190:46697
I20260621 05:13:20.269393 19169 heartbeater.cc:464] Registering TS with master...
I20260621 05:13:20.269582 19169 heartbeater.cc:511] Master 127.3.131.190:46697 requested a full tablet report, sending...
I20260621 05:13:20.270112 18978 ts_manager.cc:194] Registered new tserver with Master: af09019aeb194a60b128a035658ed606 (127.3.131.129:41035)
I20260621 05:13:20.270759 18978 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.131.129@KRBTEST.COM'} at 127.3.131.129:39453
WARNING: no policy specified for kudu/127.3.131.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.131.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.131.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782018616648578-3598-0/krb5kdc/kudu_127.3.131.130.keytab.
Entry for principal kudu/127.3.131.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782018616648578-3598-0/krb5kdc/kudu_127.3.131.130.keytab.
I20260621 05:13:20.291344 3598 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWZOx2_/build/release/bin/kudu
/tmp/dist-test-taskWZOx2_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782018616648578-3598-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782018616648578-3598-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782018616648578-3598-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782018616648578-3598-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.131.130:0
--local_ip_for_outbound_sockets=127.3.131.130
--webserver_interface=127.3.131.130
--webserver_port=0
--tserver_master_addrs=127.3.131.190:46697
--keytab_file=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782018616648578-3598-0/krb5kdc/kudu_127.3.131.130.keytab
--principal=kudu/127.3.131.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.131.148:44007
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782018616648578-3598-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782018616648578-3598-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782018616648578-3598-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 05:13:20.374058 19174 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 05:13:20.374307 19174 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 05:13:20.374346 19174 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 05:13:20.375581 19174 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 05:13:20.375654 19174 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 05:13:20.375699 19174 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.131.130
I20260621 05:13:20.377197 19174 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.131.148:44007
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782018616648578-3598-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782018616648578-3598-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782018616648578-3598-0/krb5kdc/kudu_127.3.131.130.keytab
--principal=kudu/127.3.131.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.131.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782018616648578-3598-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.131.130
--webserver_port=0
--tserver_master_addrs=127.3.131.190:46697
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19174
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.131.130
--log_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782018616648578-3598-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 21 Jun 2026 04:16:16 UTC on ccycloud.quasar-xycaha.root.comops.site
build id 4457881
I20260621 05:13:20.377480 19174 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 05:13:20.377712 19174 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 05:13:20.382795 19179 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:13:20.382994 19180 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:13:20.382795 19182 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:13:20.383141 19174 server_base.cc:1034] running on GCE node
I20260621 05:13:20.383436 19174 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 05:13:20.383666 19174 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 05:13:20.384908 19174 hybrid_clock.cc:648] HybridClock initialized: now 1782018800384864 us; error 88 us; skew 500 ppm
Jun 21 05:13:20 dist-test-slave-vrq1 krb5kdc[18933](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782018800, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.131.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 05:13:20.387707 19174 init.cc:377] Logged in from keytab as kudu/127.3.131.130@KRBTEST.COM (short username kudu)
I20260621 05:13:20.388192 19174 webserver.cc:460] Webserver started at http://127.3.131.130:34509/ using document root <none> and password file <none>
I20260621 05:13:20.388396 19174 fs_manager.cc:359] Metadata directory not provided
I20260621 05:13:20.388437 19174 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 05:13:20.388526 19174 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 05:13:20.389561 19174 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782018616648578-3598-0/minicluster-data/ts-1/data/instance:
uuid: "518697964a27412aa91818fcc615e6d5"
format_stamp: "Formatted at 2026-06-21 05:13:20 on dist-test-slave-vrq1"
server_key: "6a04f3ab56b30c3811a656e5d322d68f"
server_key_iv: "cf756f1d79126e3538eab75edec77821"
server_key_version: "encryptionkey@0"
I20260621 05:13:20.389868 19174 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782018616648578-3598-0/minicluster-data/ts-1/wal/instance:
uuid: "518697964a27412aa91818fcc615e6d5"
format_stamp: "Formatted at 2026-06-21 05:13:20 on dist-test-slave-vrq1"
server_key: "6a04f3ab56b30c3811a656e5d322d68f"
server_key_iv: "cf756f1d79126e3538eab75edec77821"
server_key_version: "encryptionkey@0"
I20260621 05:13:20.391417 19174 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260621 05:13:20.392205 19189 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 05:13:20.392554 19174 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260621 05:13:20.392670 19174 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782018616648578-3598-0/minicluster-data/ts-1/data,/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782018616648578-3598-0/minicluster-data/ts-1/wal
uuid: "518697964a27412aa91818fcc615e6d5"
format_stamp: "Formatted at 2026-06-21 05:13:20 on dist-test-slave-vrq1"
server_key: "6a04f3ab56b30c3811a656e5d322d68f"
server_key_iv: "cf756f1d79126e3538eab75edec77821"
server_key_version: "encryptionkey@0"
I20260621 05:13:20.392767 19174 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782018616648578-3598-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782018616648578-3598-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782018616648578-3598-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 05:13:20.416896 19174 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 05:13:20.419548 19174 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 05:13:20.419713 19174 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 05:13:20.419971 19174 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 05:13:20.420318 19174 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 05:13:20.420380 19174 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 05:13:20.420418 19174 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 05:13:20.420445 19174 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 05:13:20.426748 19174 rpc_server.cc:307] RPC server started. Bound to: 127.3.131.130:44723
I20260621 05:13:20.426847 19302 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.131.130:44723 every 8 connection(s)
I20260621 05:13:20.427286 19174 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782018616648578-3598-0/minicluster-data/ts-1/data/info.pb
Jun 21 05:13:20 dist-test-slave-vrq1 krb5kdc[18933](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782018800, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.131.130@KRBTEST.COM for kudu/127.3.131.190@KRBTEST.COM
I20260621 05:13:20.437119 19018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:13:20.428178 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.131.130:44467 (local address 127.3.131.190:46697)
0621 05:13:20.428262 (+ 84us) server_negotiation.cc:206] Beginning negotiation
0621 05:13:20.428264 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:13:20.428440 (+ 176us) server_negotiation.cc:407] Connection header received
0621 05:13:20.429008 (+ 568us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:13:20.429009 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:13:20.429022 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:13:20.429074 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:13:20.430078 (+ 1004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:13:20.430586 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:13:20.431677 (+ 1091us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:13:20.431869 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:13:20.431953 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:13:20.434315 (+ 2362us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:13:20.434316 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:13:20.434317 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:13:20.434326 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:13:20.435691 (+ 1365us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:13:20.436140 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:13:20.436142 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:13:20.436142 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:13:20.436203 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:13:20.436461 (+ 258us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:13:20.436462 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:13:20.436463 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:13:20.436583 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:13:20.436652 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0621 05:13:20.436940 (+ 288us) server_negotiation.cc:299] Negotiation successful
0621 05:13:20.437025 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260621 05:13:20.437274 3598 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWZOx2_/build/release/bin/kudu as pid 19174
I20260621 05:13:20.437328 19305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:13:20.428071 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.131.190:46697 (local address 127.3.131.130:44467)
0621 05:13:20.428300 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0621 05:13:20.428350 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0621 05:13:20.428890 (+ 540us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:13:20.429145 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:13:20.429149 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:13:20.429758 (+ 609us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:13:20.429966 (+ 208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:13:20.429971 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:13:20.430689 (+ 718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:13:20.430690 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:13:20.431574 (+ 884us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:13:20.431576 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:13:20.431963 (+ 387us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:13:20.431964 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:13:20.432094 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:13:20.432611 (+ 517us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 05:13:20.432619 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:13:20.434198 (+ 1579us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 05:13:20.435792 (+ 1594us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:13:20.435799 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:13:20.435801 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:13:20.436045 (+ 244us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:13:20.436270 (+ 225us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:13:20.436271 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:13:20.436271 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:13:20.436386 (+ 115us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:13:20.436657 (+ 271us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 05:13:20.436664 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 05:13:20.436802 (+ 138us) client_negotiation.cc:769] Sending connection context
0621 05:13:20.436917 (+ 115us) client_negotiation.cc:240] Negotiation successful
0621 05:13:20.437082 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":114,"threads_started":1}
I20260621 05:13:20.437387 3598 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782018616648578-3598-0/minicluster-data/ts-1/wal/instance
I20260621 05:13:20.437592 3598 external_mini_cluster.cc:1428] Setting key 402ed9817c9926123b8c7ccff908fca5
I20260621 05:13:20.437832 19303 heartbeater.cc:347] Connected to a master server at 127.3.131.190:46697
I20260621 05:13:20.437954 19303 heartbeater.cc:464] Registering TS with master...
I20260621 05:13:20.438133 19303 heartbeater.cc:511] Master 127.3.131.190:46697 requested a full tablet report, sending...
I20260621 05:13:20.438566 18978 ts_manager.cc:194] Registered new tserver with Master: 518697964a27412aa91818fcc615e6d5 (127.3.131.130:44723)
I20260621 05:13:20.439073 18978 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.131.130@KRBTEST.COM'} at 127.3.131.130:44467
WARNING: no policy specified for kudu/127.3.131.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.131.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.131.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782018616648578-3598-0/krb5kdc/kudu_127.3.131.131.keytab.
Entry for principal kudu/127.3.131.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782018616648578-3598-0/krb5kdc/kudu_127.3.131.131.keytab.
I20260621 05:13:20.464659 3598 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWZOx2_/build/release/bin/kudu
/tmp/dist-test-taskWZOx2_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782018616648578-3598-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782018616648578-3598-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782018616648578-3598-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782018616648578-3598-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.131.131:0
--local_ip_for_outbound_sockets=127.3.131.131
--webserver_interface=127.3.131.131
--webserver_port=0
--tserver_master_addrs=127.3.131.190:46697
--keytab_file=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782018616648578-3598-0/krb5kdc/kudu_127.3.131.131.keytab
--principal=kudu/127.3.131.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.131.148:44007
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782018616648578-3598-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782018616648578-3598-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782018616648578-3598-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 05:13:20.551353 19308 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 05:13:20.551558 19308 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 05:13:20.551581 19308 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 05:13:20.552768 19308 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 05:13:20.552814 19308 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 05:13:20.552841 19308 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.131.131
I20260621 05:13:20.554324 19308 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.131.148:44007
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782018616648578-3598-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782018616648578-3598-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782018616648578-3598-0/krb5kdc/kudu_127.3.131.131.keytab
--principal=kudu/127.3.131.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.131.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782018616648578-3598-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.131.131
--webserver_port=0
--tserver_master_addrs=127.3.131.190:46697
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19308
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.131.131
--log_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782018616648578-3598-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 21 Jun 2026 04:16:16 UTC on ccycloud.quasar-xycaha.root.comops.site
build id 4457881
I20260621 05:13:20.554577 19308 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 05:13:20.554808 19308 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 05:13:20.559945 19314 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:13:20.560199 19316 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:13:20.560276 19308 server_base.cc:1034] running on GCE node
W20260621 05:13:20.560292 19313 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:13:20.560604 19308 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 05:13:20.560886 19308 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 05:13:20.562053 19308 hybrid_clock.cc:648] HybridClock initialized: now 1782018800562024 us; error 51 us; skew 500 ppm
Jun 21 05:13:20 dist-test-slave-vrq1 krb5kdc[18933](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782018800, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.131.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 05:13:20.568768 19308 init.cc:377] Logged in from keytab as kudu/127.3.131.131@KRBTEST.COM (short username kudu)
I20260621 05:13:20.569311 19308 webserver.cc:460] Webserver started at http://127.3.131.131:43527/ using document root <none> and password file <none>
I20260621 05:13:20.569561 19308 fs_manager.cc:359] Metadata directory not provided
I20260621 05:13:20.569634 19308 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 05:13:20.569815 19308 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 05:13:20.570953 19308 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782018616648578-3598-0/minicluster-data/ts-2/data/instance:
uuid: "245dfac09a1a4fcba9d4ed15c2387fc8"
format_stamp: "Formatted at 2026-06-21 05:13:20 on dist-test-slave-vrq1"
server_key: "067ea0ddafaa5328ebc1fb466104a9e8"
server_key_iv: "417bfc56a463306fd6117f4085658fee"
server_key_version: "encryptionkey@0"
I20260621 05:13:20.571373 19308 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782018616648578-3598-0/minicluster-data/ts-2/wal/instance:
uuid: "245dfac09a1a4fcba9d4ed15c2387fc8"
format_stamp: "Formatted at 2026-06-21 05:13:20 on dist-test-slave-vrq1"
server_key: "067ea0ddafaa5328ebc1fb466104a9e8"
server_key_iv: "417bfc56a463306fd6117f4085658fee"
server_key_version: "encryptionkey@0"
I20260621 05:13:20.572875 19308 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260621 05:13:20.573882 19323 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 05:13:20.574208 19308 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260621 05:13:20.574349 19308 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782018616648578-3598-0/minicluster-data/ts-2/data,/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782018616648578-3598-0/minicluster-data/ts-2/wal
uuid: "245dfac09a1a4fcba9d4ed15c2387fc8"
format_stamp: "Formatted at 2026-06-21 05:13:20 on dist-test-slave-vrq1"
server_key: "067ea0ddafaa5328ebc1fb466104a9e8"
server_key_iv: "417bfc56a463306fd6117f4085658fee"
server_key_version: "encryptionkey@0"
I20260621 05:13:20.574441 19308 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782018616648578-3598-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782018616648578-3598-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782018616648578-3598-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 05:13:20.592132 19308 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 05:13:20.594400 19308 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 05:13:20.594563 19308 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 05:13:20.594950 19308 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 05:13:20.595477 19308 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 05:13:20.595535 19308 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 05:13:20.595559 19308 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 05:13:20.595572 19308 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 05:13:20.601754 19308 rpc_server.cc:307] RPC server started. Bound to: 127.3.131.131:38877
I20260621 05:13:20.601797 19436 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.131.131:38877 every 8 connection(s)
I20260621 05:13:20.602262 19308 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782018616648578-3598-0/minicluster-data/ts-2/data/info.pb
Jun 21 05:13:20 dist-test-slave-vrq1 krb5kdc[18933](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782018800, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.131.131@KRBTEST.COM for kudu/127.3.131.190@KRBTEST.COM
I20260621 05:13:20.610498 3598 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWZOx2_/build/release/bin/kudu as pid 19308
I20260621 05:13:20.610586 3598 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782018616648578-3598-0/minicluster-data/ts-2/wal/instance
I20260621 05:13:20.610839 3598 external_mini_cluster.cc:1428] Setting key 2c548af785807902c1ebd16c4b2e83c2
I20260621 05:13:20.613245 19018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:13:20.603128 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.131.131:52829 (local address 127.3.131.190:46697)
0621 05:13:20.603248 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0621 05:13:20.603250 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:13:20.603429 (+ 179us) server_negotiation.cc:407] Connection header received
0621 05:13:20.603886 (+ 457us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:13:20.603887 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:13:20.603896 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:13:20.603953 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:13:20.604900 (+ 947us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:13:20.605553 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:13:20.607041 (+ 1488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:13:20.607363 (+ 322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:13:20.607426 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:13:20.609824 (+ 2398us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:13:20.609826 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:13:20.609826 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:13:20.609835 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:13:20.611793 (+ 1958us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:13:20.612236 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:13:20.612237 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:13:20.612238 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:13:20.612298 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:13:20.612577 (+ 279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:13:20.612578 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:13:20.612578 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:13:20.612703 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:13:20.612774 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0621 05:13:20.613068 (+ 294us) server_negotiation.cc:299] Negotiation successful
0621 05:13:20.613151 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260621 05:13:20.613478 19439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:13:20.603097 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.131.190:46697 (local address 127.3.131.131:52829)
0621 05:13:20.603292 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0621 05:13:20.603335 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0621 05:13:20.603806 (+ 471us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:13:20.604007 (+ 201us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:13:20.604010 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:13:20.604590 (+ 580us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:13:20.604802 (+ 212us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:13:20.604807 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:13:20.605653 (+ 846us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:13:20.605654 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:13:20.606686 (+ 1032us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:13:20.606689 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:13:20.607454 (+ 765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:13:20.607455 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:13:20.607574 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:13:20.608117 (+ 543us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 05:13:20.608126 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:13:20.609718 (+ 1592us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 05:13:20.611912 (+ 2194us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:13:20.611918 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:13:20.611921 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:13:20.612153 (+ 232us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:13:20.612384 (+ 231us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:13:20.612385 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:13:20.612386 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:13:20.612500 (+ 114us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:13:20.612776 (+ 276us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 05:13:20.612783 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 05:13:20.612929 (+ 146us) client_negotiation.cc:769] Sending connection context
0621 05:13:20.613036 (+ 107us) client_negotiation.cc:240] Negotiation successful
0621 05:13:20.613207 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":154,"thread_start_us":113,"threads_started":1}
I20260621 05:13:20.614200 19437 heartbeater.cc:347] Connected to a master server at 127.3.131.190:46697
I20260621 05:13:20.614310 19437 heartbeater.cc:464] Registering TS with master...
I20260621 05:13:20.614676 19437 heartbeater.cc:511] Master 127.3.131.190:46697 requested a full tablet report, sending...
I20260621 05:13:20.615388 18978 ts_manager.cc:194] Registered new tserver with Master: 245dfac09a1a4fcba9d4ed15c2387fc8 (127.3.131.131:38877)
I20260621 05:13:20.615926 18978 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.131.131@KRBTEST.COM'} at 127.3.131.131:52829
I20260621 05:13:20.623400 3598 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260621 05:13:20.632117 19018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:13:20.624987 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39116 (local address 127.3.131.190:46697)
0621 05:13:20.625087 (+ 100us) server_negotiation.cc:206] Beginning negotiation
0621 05:13:20.625090 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0621 05:13:20.625197 (+ 107us) server_negotiation.cc:407] Connection header received
0621 05:13:20.625339 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:13:20.625341 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:13:20.625352 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:13:20.625427 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:13:20.626293 (+ 866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:13:20.626801 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:13:20.627616 (+ 815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:13:20.627818 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:13:20.627905 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:13:20.629246 (+ 1341us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:13:20.629249 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:13:20.629249 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:13:20.629258 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:13:20.630717 (+ 1459us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:13:20.631132 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:13:20.631134 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:13:20.631134 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:13:20.631257 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:13:20.631465 (+ 208us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:13:20.631466 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:13:20.631466 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:13:20.631635 (+ 169us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:13:20.631744 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0621 05:13:20.631917 (+ 173us) server_negotiation.cc:299] Negotiation successful
0621 05:13:20.632005 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260621 05:13:20.634318 18978 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39116:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260621 05:13:20.636157 18978 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260621 05:13:20.644106 19449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:13:20.639234 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50648 (local address 127.3.131.131:38877)
0621 05:13:20.639539 (+ 305us) server_negotiation.cc:206] Beginning negotiation
0621 05:13:20.639545 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 05:13:20.639820 (+ 275us) server_negotiation.cc:407] Connection header received
0621 05:13:20.640208 (+ 388us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:13:20.640211 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:13:20.640229 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:13:20.640284 (+ 55us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 05:13:20.640591 (+ 307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:13:20.641282 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:13:20.642825 (+ 1543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:13:20.643566 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:13:20.643669 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:13:20.643736 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0621 05:13:20.643940 (+ 204us) server_negotiation.cc:299] Negotiation successful
0621 05:13:20.643995 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":267,"thread_start_us":231,"threads_started":1}
I20260621 05:13:20.644121 19448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:13:20.639133 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.131.129:41035 (local address 127.0.0.1:38340)
0621 05:13:20.639333 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0621 05:13:20.639384 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0621 05:13:20.639464 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:13:20.639891 (+ 427us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:13:20.639895 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:13:20.639898 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 05:13:20.640174 (+ 276us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:13:20.640178 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:13:20.641671 (+ 1493us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:13:20.641672 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:13:20.642732 (+ 1060us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:13:20.642734 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:13:20.643780 (+ 1046us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:13:20.643781 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:13:20.643862 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:13:20.643867 (+ 5us) client_negotiation.cc:769] Sending connection context
0621 05:13:20.643921 (+ 54us) client_negotiation.cc:240] Negotiation successful
0621 05:13:20.644015 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":164,"thread_start_us":94,"threads_started":1}
I20260621 05:13:20.644520 19450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:13:20.639268 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38340 (local address 127.3.131.129:41035)
0621 05:13:20.639563 (+ 295us) server_negotiation.cc:206] Beginning negotiation
0621 05:13:20.639569 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 05:13:20.639587 (+ 18us) server_negotiation.cc:407] Connection header received
0621 05:13:20.639636 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:13:20.639639 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:13:20.639657 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:13:20.639751 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 05:13:20.640289 (+ 538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:13:20.641561 (+ 1272us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:13:20.642886 (+ 1325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:13:20.643612 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:13:20.643687 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:13:20.643740 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0621 05:13:20.644391 (+ 651us) server_negotiation.cc:299] Negotiation successful
0621 05:13:20.644440 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":237,"thread_start_us":102,"threads_started":1}
I20260621 05:13:20.644529 19452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:13:20.639788 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.131.130:44723 (local address 127.0.0.1:56448)
0621 05:13:20.640600 (+ 812us) negotiation.cc:107] Waiting for socket to connect
0621 05:13:20.640613 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0621 05:13:20.640696 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:13:20.640993 (+ 297us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:13:20.640994 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:13:20.640995 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 05:13:20.641041 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:13:20.641043 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:13:20.642199 (+ 1156us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:13:20.642200 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:13:20.643167 (+ 967us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:13:20.643170 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:13:20.644204 (+ 1034us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:13:20.644205 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:13:20.644330 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:13:20.644334 (+ 4us) client_negotiation.cc:769] Sending connection context
0621 05:13:20.644396 (+ 62us) client_negotiation.cc:240] Negotiation successful
0621 05:13:20.644443 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":786,"thread_start_us":70,"threads_started":1}
I20260621 05:13:20.644124 19451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:13:20.639133 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.131.131:38877 (local address 127.0.0.1:50648)
0621 05:13:20.639746 (+ 613us) negotiation.cc:107] Waiting for socket to connect
0621 05:13:20.639762 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0621 05:13:20.640085 (+ 323us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:13:20.640438 (+ 353us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:13:20.640439 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:13:20.640441 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 05:13:20.640500 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:13:20.640502 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:13:20.641397 (+ 895us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:13:20.641398 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:13:20.642709 (+ 1311us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:13:20.642713 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:13:20.643672 (+ 959us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:13:20.643673 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:13:20.643795 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:13:20.643810 (+ 15us) client_negotiation.cc:769] Sending connection context
0621 05:13:20.643921 (+ 111us) client_negotiation.cc:240] Negotiation successful
0621 05:13:20.644024 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":587,"mutex_wait_us":444,"thread_start_us":129,"threads_started":1}
I20260621 05:13:20.645450 19453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:13:20.640263 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56448 (local address 127.3.131.130:44723)
0621 05:13:20.640542 (+ 279us) server_negotiation.cc:206] Beginning negotiation
0621 05:13:20.640547 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 05:13:20.640769 (+ 222us) server_negotiation.cc:407] Connection header received
0621 05:13:20.640820 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:13:20.640823 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:13:20.640839 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:13:20.640913 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 05:13:20.641124 (+ 211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:13:20.642082 (+ 958us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:13:20.643290 (+ 1208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:13:20.644133 (+ 843us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:13:20.645211 (+ 1078us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:13:20.645286 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0621 05:13:20.645314 (+ 28us) server_negotiation.cc:299] Negotiation successful
0621 05:13:20.645363 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":238,"thread_start_us":206,"threads_started":1}
I20260621 05:13:20.645481 19371 tablet_service.cc:1467] Processing CreateTablet for tablet c51ff510612f4d59ba97736d257d4296 (DEFAULT_TABLE table=test-table [id=2dd763ba8d364a4b9eec4fe8548c9bd9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 05:13:20.645470 19103 tablet_service.cc:1467] Processing CreateTablet for tablet c51ff510612f4d59ba97736d257d4296 (DEFAULT_TABLE table=test-table [id=2dd763ba8d364a4b9eec4fe8548c9bd9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 05:13:20.645876 19371 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c51ff510612f4d59ba97736d257d4296. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 05:13:20.645875 19103 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c51ff510612f4d59ba97736d257d4296. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 05:13:20.647987 19237 tablet_service.cc:1467] Processing CreateTablet for tablet c51ff510612f4d59ba97736d257d4296 (DEFAULT_TABLE table=test-table [id=2dd763ba8d364a4b9eec4fe8548c9bd9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 05:13:20.648453 19237 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c51ff510612f4d59ba97736d257d4296. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 05:13:20.649151 19454 tablet_bootstrap.cc:492] T c51ff510612f4d59ba97736d257d4296 P 245dfac09a1a4fcba9d4ed15c2387fc8: Bootstrap starting.
I20260621 05:13:20.649734 19454 tablet_bootstrap.cc:654] T c51ff510612f4d59ba97736d257d4296 P 245dfac09a1a4fcba9d4ed15c2387fc8: Neither blocks nor log segments found. Creating new log.
I20260621 05:13:20.650225 19454 log.cc:824] T c51ff510612f4d59ba97736d257d4296 P 245dfac09a1a4fcba9d4ed15c2387fc8: Log is configured to *not* fsync() on all Append() calls
I20260621 05:13:20.650755 19456 tablet_bootstrap.cc:492] T c51ff510612f4d59ba97736d257d4296 P af09019aeb194a60b128a035658ed606: Bootstrap starting.
I20260621 05:13:20.651403 19454 tablet_bootstrap.cc:492] T c51ff510612f4d59ba97736d257d4296 P 245dfac09a1a4fcba9d4ed15c2387fc8: No bootstrap required, opened a new log
I20260621 05:13:20.651481 19454 ts_tablet_manager.cc:1397] T c51ff510612f4d59ba97736d257d4296 P 245dfac09a1a4fcba9d4ed15c2387fc8: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260621 05:13:20.651652 19456 tablet_bootstrap.cc:654] T c51ff510612f4d59ba97736d257d4296 P af09019aeb194a60b128a035658ed606: Neither blocks nor log segments found. Creating new log.
I20260621 05:13:20.652382 19456 log.cc:824] T c51ff510612f4d59ba97736d257d4296 P af09019aeb194a60b128a035658ed606: Log is configured to *not* fsync() on all Append() calls
I20260621 05:13:20.652835 19457 tablet_bootstrap.cc:492] T c51ff510612f4d59ba97736d257d4296 P 518697964a27412aa91818fcc615e6d5: Bootstrap starting.
I20260621 05:13:20.653435 19457 tablet_bootstrap.cc:654] T c51ff510612f4d59ba97736d257d4296 P 518697964a27412aa91818fcc615e6d5: Neither blocks nor log segments found. Creating new log.
I20260621 05:13:20.653961 19454 raft_consensus.cc:348] T c51ff510612f4d59ba97736d257d4296 P 245dfac09a1a4fcba9d4ed15c2387fc8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "245dfac09a1a4fcba9d4ed15c2387fc8" member_type: VOTER last_known_addr { host: "127.3.131.131" port: 38877 } } peers { permanent_uuid: "518697964a27412aa91818fcc615e6d5" member_type: VOTER last_known_addr { host: "127.3.131.130" port: 44723 } } peers { permanent_uuid: "af09019aeb194a60b128a035658ed606" member_type: VOTER last_known_addr { host: "127.3.131.129" port: 41035 } }
I20260621 05:13:20.654145 19454 raft_consensus.cc:374] T c51ff510612f4d59ba97736d257d4296 P 245dfac09a1a4fcba9d4ed15c2387fc8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 05:13:20.654192 19454 raft_consensus.cc:729] T c51ff510612f4d59ba97736d257d4296 P 245dfac09a1a4fcba9d4ed15c2387fc8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 245dfac09a1a4fcba9d4ed15c2387fc8, State: Initialized, Role: FOLLOWER
I20260621 05:13:20.654320 19454 consensus_queue.cc:260] T c51ff510612f4d59ba97736d257d4296 P 245dfac09a1a4fcba9d4ed15c2387fc8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "245dfac09a1a4fcba9d4ed15c2387fc8" member_type: VOTER last_known_addr { host: "127.3.131.131" port: 38877 } } peers { permanent_uuid: "518697964a27412aa91818fcc615e6d5" member_type: VOTER last_known_addr { host: "127.3.131.130" port: 44723 } } peers { permanent_uuid: "af09019aeb194a60b128a035658ed606" member_type: VOTER last_known_addr { host: "127.3.131.129" port: 41035 } }
I20260621 05:13:20.654347 19457 log.cc:824] T c51ff510612f4d59ba97736d257d4296 P 518697964a27412aa91818fcc615e6d5: Log is configured to *not* fsync() on all Append() calls
I20260621 05:13:20.654660 19454 ts_tablet_manager.cc:1428] T c51ff510612f4d59ba97736d257d4296 P 245dfac09a1a4fcba9d4ed15c2387fc8: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260621 05:13:20.654708 19437 heartbeater.cc:503] Master 127.3.131.190:46697 was elected leader, sending a full tablet report...
I20260621 05:13:20.655421 19456 tablet_bootstrap.cc:492] T c51ff510612f4d59ba97736d257d4296 P af09019aeb194a60b128a035658ed606: No bootstrap required, opened a new log
I20260621 05:13:20.655521 19456 ts_tablet_manager.cc:1397] T c51ff510612f4d59ba97736d257d4296 P af09019aeb194a60b128a035658ed606: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20260621 05:13:20.655550 19457 tablet_bootstrap.cc:492] T c51ff510612f4d59ba97736d257d4296 P 518697964a27412aa91818fcc615e6d5: No bootstrap required, opened a new log
I20260621 05:13:20.655625 19457 ts_tablet_manager.cc:1397] T c51ff510612f4d59ba97736d257d4296 P 518697964a27412aa91818fcc615e6d5: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260621 05:13:20.657389 19456 raft_consensus.cc:348] T c51ff510612f4d59ba97736d257d4296 P af09019aeb194a60b128a035658ed606 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "245dfac09a1a4fcba9d4ed15c2387fc8" member_type: VOTER last_known_addr { host: "127.3.131.131" port: 38877 } } peers { permanent_uuid: "518697964a27412aa91818fcc615e6d5" member_type: VOTER last_known_addr { host: "127.3.131.130" port: 44723 } } peers { permanent_uuid: "af09019aeb194a60b128a035658ed606" member_type: VOTER last_known_addr { host: "127.3.131.129" port: 41035 } }
I20260621 05:13:20.657578 19456 raft_consensus.cc:374] T c51ff510612f4d59ba97736d257d4296 P af09019aeb194a60b128a035658ed606 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 05:13:20.657626 19456 raft_consensus.cc:729] T c51ff510612f4d59ba97736d257d4296 P af09019aeb194a60b128a035658ed606 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af09019aeb194a60b128a035658ed606, State: Initialized, Role: FOLLOWER
I20260621 05:13:20.657765 19456 consensus_queue.cc:260] T c51ff510612f4d59ba97736d257d4296 P af09019aeb194a60b128a035658ed606 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "245dfac09a1a4fcba9d4ed15c2387fc8" member_type: VOTER last_known_addr { host: "127.3.131.131" port: 38877 } } peers { permanent_uuid: "518697964a27412aa91818fcc615e6d5" member_type: VOTER last_known_addr { host: "127.3.131.130" port: 44723 } } peers { permanent_uuid: "af09019aeb194a60b128a035658ed606" member_type: VOTER last_known_addr { host: "127.3.131.129" port: 41035 } }
I20260621 05:13:20.658170 19169 heartbeater.cc:503] Master 127.3.131.190:46697 was elected leader, sending a full tablet report...
I20260621 05:13:20.658125 19456 ts_tablet_manager.cc:1428] T c51ff510612f4d59ba97736d257d4296 P af09019aeb194a60b128a035658ed606: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260621 05:13:20.658059 19457 raft_consensus.cc:348] T c51ff510612f4d59ba97736d257d4296 P 518697964a27412aa91818fcc615e6d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "245dfac09a1a4fcba9d4ed15c2387fc8" member_type: VOTER last_known_addr { host: "127.3.131.131" port: 38877 } } peers { permanent_uuid: "518697964a27412aa91818fcc615e6d5" member_type: VOTER last_known_addr { host: "127.3.131.130" port: 44723 } } peers { permanent_uuid: "af09019aeb194a60b128a035658ed606" member_type: VOTER last_known_addr { host: "127.3.131.129" port: 41035 } }
I20260621 05:13:20.658250 19457 raft_consensus.cc:374] T c51ff510612f4d59ba97736d257d4296 P 518697964a27412aa91818fcc615e6d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 05:13:20.658286 19457 raft_consensus.cc:729] T c51ff510612f4d59ba97736d257d4296 P 518697964a27412aa91818fcc615e6d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 518697964a27412aa91818fcc615e6d5, State: Initialized, Role: FOLLOWER
I20260621 05:13:20.658430 19457 consensus_queue.cc:260] T c51ff510612f4d59ba97736d257d4296 P 518697964a27412aa91818fcc615e6d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "245dfac09a1a4fcba9d4ed15c2387fc8" member_type: VOTER last_known_addr { host: "127.3.131.131" port: 38877 } } peers { permanent_uuid: "518697964a27412aa91818fcc615e6d5" member_type: VOTER last_known_addr { host: "127.3.131.130" port: 44723 } } peers { permanent_uuid: "af09019aeb194a60b128a035658ed606" member_type: VOTER last_known_addr { host: "127.3.131.129" port: 41035 } }
I20260621 05:13:20.658764 19457 ts_tablet_manager.cc:1428] T c51ff510612f4d59ba97736d257d4296 P 518697964a27412aa91818fcc615e6d5: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260621 05:13:20.658790 19303 heartbeater.cc:503] Master 127.3.131.190:46697 was elected leader, sending a full tablet report...
W20260621 05:13:20.677908 19304 tablet.cc:2369] T c51ff510612f4d59ba97736d257d4296 P 518697964a27412aa91818fcc615e6d5: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260621 05:13:20.689987 19459 raft_consensus.cc:482] T c51ff510612f4d59ba97736d257d4296 P 245dfac09a1a4fcba9d4ed15c2387fc8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260621 05:13:20.690187 19459 raft_consensus.cc:504] T c51ff510612f4d59ba97736d257d4296 P 245dfac09a1a4fcba9d4ed15c2387fc8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "245dfac09a1a4fcba9d4ed15c2387fc8" member_type: VOTER last_known_addr { host: "127.3.131.131" port: 38877 } } peers { permanent_uuid: "518697964a27412aa91818fcc615e6d5" member_type: VOTER last_known_addr { host: "127.3.131.130" port: 44723 } } peers { permanent_uuid: "af09019aeb194a60b128a035658ed606" member_type: VOTER last_known_addr { host: "127.3.131.129" port: 41035 } }
I20260621 05:13:20.690647 19459 leader_election.cc:288] T c51ff510612f4d59ba97736d257d4296 P 245dfac09a1a4fcba9d4ed15c2387fc8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 518697964a27412aa91818fcc615e6d5 (127.3.131.130:44723), af09019aeb194a60b128a035658ed606 (127.3.131.129:41035)
I20260621 05:13:20.693953 19450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:13:20.690986 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.131.131:37795 (local address 127.3.131.129:41035)
0621 05:13:20.691042 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0621 05:13:20.691044 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:13:20.691247 (+ 203us) server_negotiation.cc:407] Connection header received
0621 05:13:20.691288 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:13:20.691289 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:13:20.691300 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:13:20.691371 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 05:13:20.691565 (+ 194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:13:20.692051 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:13:20.693155 (+ 1104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:13:20.693562 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:13:20.693652 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:13:20.693662 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0621 05:13:20.693828 (+ 166us) server_negotiation.cc:299] Negotiation successful
0621 05:13:20.693874 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260621 05:13:20.693967 19439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:13:20.690868 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.131.130:44723 (local address 127.3.131.131:43275)
0621 05:13:20.690960 (+ 92us) negotiation.cc:107] Waiting for socket to connect
0621 05:13:20.690973 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0621 05:13:20.691022 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:13:20.691309 (+ 287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:13:20.691310 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:13:20.691312 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 05:13:20.691395 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:13:20.691398 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:13:20.692204 (+ 806us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:13:20.692205 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:13:20.693000 (+ 795us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:13:20.693002 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:13:20.693653 (+ 651us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:13:20.693654 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:13:20.693760 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:13:20.693763 (+ 3us) client_negotiation.cc:769] Sending connection context
0621 05:13:20.693843 (+ 80us) client_negotiation.cc:240] Negotiation successful
0621 05:13:20.693880 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":70,"mutex_wait_us":5}
I20260621 05:13:20.693979 19453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:13:20.690959 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.131.131:43275 (local address 127.3.131.130:44723)
0621 05:13:20.691009 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0621 05:13:20.691011 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:13:20.691024 (+ 13us) server_negotiation.cc:407] Connection header received
0621 05:13:20.691090 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:13:20.691091 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:13:20.691102 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:13:20.691212 (+ 110us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 05:13:20.691472 (+ 260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:13:20.692086 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:13:20.693103 (+ 1017us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:13:20.693557 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:13:20.693656 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:13:20.693666 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0621 05:13:20.693867 (+ 201us) server_negotiation.cc:299] Negotiation successful
0621 05:13:20.693908 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260621 05:13:20.693987 19463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:13:20.690899 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.131.129:41035 (local address 127.3.131.131:37795)
0621 05:13:20.691127 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0621 05:13:20.691138 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0621 05:13:20.691231 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:13:20.691388 (+ 157us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:13:20.691389 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:13:20.691391 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 05:13:20.691475 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:13:20.691476 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:13:20.692150 (+ 674us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:13:20.692151 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:13:20.693068 (+ 917us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:13:20.693071 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:13:20.693652 (+ 581us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:13:20.693653 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:13:20.693743 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:13:20.693747 (+ 4us) client_negotiation.cc:769] Sending connection context
0621 05:13:20.693795 (+ 48us) client_negotiation.cc:240] Negotiation successful
0621 05:13:20.693867 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":208,"mutex_wait_us":16,"thread_start_us":150,"threads_started":1}
I20260621 05:13:20.694522 19257 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c51ff510612f4d59ba97736d257d4296" candidate_uuid: "245dfac09a1a4fcba9d4ed15c2387fc8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "518697964a27412aa91818fcc615e6d5" is_pre_election: true
I20260621 05:13:20.694531 19123 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c51ff510612f4d59ba97736d257d4296" candidate_uuid: "245dfac09a1a4fcba9d4ed15c2387fc8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af09019aeb194a60b128a035658ed606" is_pre_election: true
I20260621 05:13:20.694720 19257 raft_consensus.cc:2454] T c51ff510612f4d59ba97736d257d4296 P 518697964a27412aa91818fcc615e6d5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 245dfac09a1a4fcba9d4ed15c2387fc8 in term 0.
I20260621 05:13:20.694739 19123 raft_consensus.cc:2454] T c51ff510612f4d59ba97736d257d4296 P af09019aeb194a60b128a035658ed606 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 245dfac09a1a4fcba9d4ed15c2387fc8 in term 0.
I20260621 05:13:20.695022 19324 leader_election.cc:302] T c51ff510612f4d59ba97736d257d4296 P 245dfac09a1a4fcba9d4ed15c2387fc8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 245dfac09a1a4fcba9d4ed15c2387fc8, 518697964a27412aa91818fcc615e6d5; no voters:
I20260621 05:13:20.695200 19459 raft_consensus.cc:2781] T c51ff510612f4d59ba97736d257d4296 P 245dfac09a1a4fcba9d4ed15c2387fc8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260621 05:13:20.695286 19459 raft_consensus.cc:482] T c51ff510612f4d59ba97736d257d4296 P 245dfac09a1a4fcba9d4ed15c2387fc8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260621 05:13:20.695320 19459 raft_consensus.cc:3037] T c51ff510612f4d59ba97736d257d4296 P 245dfac09a1a4fcba9d4ed15c2387fc8 [term 0 FOLLOWER]: Advancing to term 1
I20260621 05:13:20.696163 19459 raft_consensus.cc:504] T c51ff510612f4d59ba97736d257d4296 P 245dfac09a1a4fcba9d4ed15c2387fc8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "245dfac09a1a4fcba9d4ed15c2387fc8" member_type: VOTER last_known_addr { host: "127.3.131.131" port: 38877 } } peers { permanent_uuid: "518697964a27412aa91818fcc615e6d5" member_type: VOTER last_known_addr { host: "127.3.131.130" port: 44723 } } peers { permanent_uuid: "af09019aeb194a60b128a035658ed606" member_type: VOTER last_known_addr { host: "127.3.131.129" port: 41035 } }
I20260621 05:13:20.696347 19459 leader_election.cc:288] T c51ff510612f4d59ba97736d257d4296 P 245dfac09a1a4fcba9d4ed15c2387fc8 [CANDIDATE]: Term 1 election: Requested vote from peers 518697964a27412aa91818fcc615e6d5 (127.3.131.130:44723), af09019aeb194a60b128a035658ed606 (127.3.131.129:41035)
I20260621 05:13:20.696568 19123 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c51ff510612f4d59ba97736d257d4296" candidate_uuid: "245dfac09a1a4fcba9d4ed15c2387fc8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af09019aeb194a60b128a035658ed606"
I20260621 05:13:20.696568 19257 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c51ff510612f4d59ba97736d257d4296" candidate_uuid: "245dfac09a1a4fcba9d4ed15c2387fc8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "518697964a27412aa91818fcc615e6d5"
I20260621 05:13:20.696667 19257 raft_consensus.cc:3037] T c51ff510612f4d59ba97736d257d4296 P 518697964a27412aa91818fcc615e6d5 [term 0 FOLLOWER]: Advancing to term 1
I20260621 05:13:20.696669 19123 raft_consensus.cc:3037] T c51ff510612f4d59ba97736d257d4296 P af09019aeb194a60b128a035658ed606 [term 0 FOLLOWER]: Advancing to term 1
I20260621 05:13:20.697638 19257 raft_consensus.cc:2454] T c51ff510612f4d59ba97736d257d4296 P 518697964a27412aa91818fcc615e6d5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 245dfac09a1a4fcba9d4ed15c2387fc8 in term 1.
I20260621 05:13:20.697638 19123 raft_consensus.cc:2454] T c51ff510612f4d59ba97736d257d4296 P af09019aeb194a60b128a035658ed606 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 245dfac09a1a4fcba9d4ed15c2387fc8 in term 1.
I20260621 05:13:20.697842 19324 leader_election.cc:302] T c51ff510612f4d59ba97736d257d4296 P 245dfac09a1a4fcba9d4ed15c2387fc8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 245dfac09a1a4fcba9d4ed15c2387fc8, 518697964a27412aa91818fcc615e6d5; no voters:
I20260621 05:13:20.697968 19459 raft_consensus.cc:2781] T c51ff510612f4d59ba97736d257d4296 P 245dfac09a1a4fcba9d4ed15c2387fc8 [term 1 FOLLOWER]: Leader election won for term 1
I20260621 05:13:20.698164 19459 raft_consensus.cc:686] T c51ff510612f4d59ba97736d257d4296 P 245dfac09a1a4fcba9d4ed15c2387fc8 [term 1 LEADER]: Becoming Leader. State: Replica: 245dfac09a1a4fcba9d4ed15c2387fc8, State: Running, Role: LEADER
I20260621 05:13:20.698354 19459 consensus_queue.cc:237] T c51ff510612f4d59ba97736d257d4296 P 245dfac09a1a4fcba9d4ed15c2387fc8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "245dfac09a1a4fcba9d4ed15c2387fc8" member_type: VOTER last_known_addr { host: "127.3.131.131" port: 38877 } } peers { permanent_uuid: "518697964a27412aa91818fcc615e6d5" member_type: VOTER last_known_addr { host: "127.3.131.130" port: 44723 } } peers { permanent_uuid: "af09019aeb194a60b128a035658ed606" member_type: VOTER last_known_addr { host: "127.3.131.129" port: 41035 } }
I20260621 05:13:20.699414 18978 catalog_manager.cc:5495] T c51ff510612f4d59ba97736d257d4296 P 245dfac09a1a4fcba9d4ed15c2387fc8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 245dfac09a1a4fcba9d4ed15c2387fc8 (127.3.131.131). New cstate: current_term: 1 leader_uuid: "245dfac09a1a4fcba9d4ed15c2387fc8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "245dfac09a1a4fcba9d4ed15c2387fc8" member_type: VOTER last_known_addr { host: "127.3.131.131" port: 38877 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "518697964a27412aa91818fcc615e6d5" member_type: VOTER last_known_addr { host: "127.3.131.130" port: 44723 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "af09019aeb194a60b128a035658ed606" member_type: VOTER last_known_addr { host: "127.3.131.129" port: 41035 } health_report { overall_health: UNKNOWN } } }
I20260621 05:13:20.720606 19449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:13:20.717877 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50654 (local address 127.3.131.131:38877)
0621 05:13:20.717946 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0621 05:13:20.717948 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:13:20.717958 (+ 10us) server_negotiation.cc:407] Connection header received
0621 05:13:20.717997 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:13:20.717998 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:13:20.718015 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:13:20.718076 (+ 61us) server_negotiation.cc:226] Negotiated authn=TOKEN
0621 05:13:20.718202 (+ 126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:13:20.718685 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:13:20.719672 (+ 987us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:13:20.719882 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:13:20.719948 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:13:20.720172 (+ 224us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0621 05:13:20.720292 (+ 120us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0621 05:13:20.720394 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0621 05:13:20.720471 (+ 77us) server_negotiation.cc:299] Negotiation successful
0621 05:13:20.720494 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260621 05:13:20.722811 19123 raft_consensus.cc:1261] T c51ff510612f4d59ba97736d257d4296 P af09019aeb194a60b128a035658ed606 [term 1 FOLLOWER]: Refusing update from remote peer 245dfac09a1a4fcba9d4ed15c2387fc8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 05:13:20.722811 19257 raft_consensus.cc:1261] T c51ff510612f4d59ba97736d257d4296 P 518697964a27412aa91818fcc615e6d5 [term 1 FOLLOWER]: Refusing update from remote peer 245dfac09a1a4fcba9d4ed15c2387fc8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 05:13:20.723251 19459 consensus_queue.cc:1035] T c51ff510612f4d59ba97736d257d4296 P 245dfac09a1a4fcba9d4ed15c2387fc8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "af09019aeb194a60b128a035658ed606" member_type: VOTER last_known_addr { host: "127.3.131.129" port: 41035 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 05:13:20.723407 19464 consensus_queue.cc:1035] T c51ff510612f4d59ba97736d257d4296 P 245dfac09a1a4fcba9d4ed15c2387fc8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "518697964a27412aa91818fcc615e6d5" member_type: VOTER last_known_addr { host: "127.3.131.130" port: 44723 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 05:13:20.726181 19467 mvcc.cc:205] Tried to move back new op lower bound from 7299149007756890112 to 7299149007661355008. Current Snapshot: MvccSnapshot[applied={T|T < 7299149007756890112}]
I20260621 05:13:20.726927 19468 mvcc.cc:205] Tried to move back new op lower bound from 7299149007756890112 to 7299149007661355008. Current Snapshot: MvccSnapshot[applied={T|T < 7299149007756890112 or (T in {7299149007756890112})}]
I20260621 05:13:20.727277 19469 mvcc.cc:205] Tried to move back new op lower bound from 7299149007756890112 to 7299149007661355008. Current Snapshot: MvccSnapshot[applied={T|T < 7299149007756890112}]
I20260621 05:13:20.728559 18978 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39116:
table { table_name: "test-table" } modify_external_catalogs: true
I20260621 05:13:20.728650 18978 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39116:
table { table_name: "test-table" } modify_external_catalogs: true
I20260621 05:13:20.729601 18978 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 5b3269e323af422bb256795282d54866: Sending DeleteTablet for 3 replicas of tablet c51ff510612f4d59ba97736d257d4296
I20260621 05:13:20.730140 19237 tablet_service.cc:1514] Processing DeleteTablet for tablet c51ff510612f4d59ba97736d257d4296 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 05:13:20 UTC) from {username='kudu', principal='kudu/127.3.131.190@KRBTEST.COM'} at 127.0.0.1:56448
I20260621 05:13:20.730160 19103 tablet_service.cc:1514] Processing DeleteTablet for tablet c51ff510612f4d59ba97736d257d4296 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 05:13:20 UTC) from {username='kudu', principal='kudu/127.3.131.190@KRBTEST.COM'} at 127.0.0.1:38340
I20260621 05:13:20.730172 19371 tablet_service.cc:1514] Processing DeleteTablet for tablet c51ff510612f4d59ba97736d257d4296 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 05:13:20 UTC) from {username='kudu', principal='kudu/127.3.131.190@KRBTEST.COM'} at 127.0.0.1:50648
I20260621 05:13:20.730360 19477 tablet_replica.cc:330] T c51ff510612f4d59ba97736d257d4296 P af09019aeb194a60b128a035658ed606: stopping tablet replica
I20260621 05:13:20.730466 19477 raft_consensus.cc:2229] T c51ff510612f4d59ba97736d257d4296 P af09019aeb194a60b128a035658ed606 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260621 05:13:20.730470 3598 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWZOx2_/build/release/bin/kudu with pid 19040
I20260621 05:13:20.730597 19477 raft_consensus.cc:2258] T c51ff510612f4d59ba97736d257d4296 P af09019aeb194a60b128a035658ed606 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 05:13:20.735366 19479 tablet_replica.cc:330] T c51ff510612f4d59ba97736d257d4296 P 245dfac09a1a4fcba9d4ed15c2387fc8: stopping tablet replica
I20260621 05:13:20.735548 19479 raft_consensus.cc:2229] T c51ff510612f4d59ba97736d257d4296 P 245dfac09a1a4fcba9d4ed15c2387fc8 [term 1 LEADER]: Raft consensus shutting down.
I20260621 05:13:20.735713 19479 raft_consensus.cc:2258] T c51ff510612f4d59ba97736d257d4296 P 245dfac09a1a4fcba9d4ed15c2387fc8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 05:13:20.735713 19478 tablet_replica.cc:330] T c51ff510612f4d59ba97736d257d4296 P 518697964a27412aa91818fcc615e6d5: stopping tablet replica
I20260621 05:13:20.735935 19478 raft_consensus.cc:2229] T c51ff510612f4d59ba97736d257d4296 P 518697964a27412aa91818fcc615e6d5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260621 05:13:20.736100 19478 raft_consensus.cc:2258] T c51ff510612f4d59ba97736d257d4296 P 518697964a27412aa91818fcc615e6d5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 05:13:20.736303 19479 ts_tablet_manager.cc:1905] T c51ff510612f4d59ba97736d257d4296 P 245dfac09a1a4fcba9d4ed15c2387fc8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 05:13:20.736655 19478 ts_tablet_manager.cc:1905] T c51ff510612f4d59ba97736d257d4296 P 518697964a27412aa91818fcc615e6d5: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260621 05:13:20.737212 18965 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.3.131.129:41035 (error 108)
I20260621 05:13:20.737851 3598 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWZOx2_/build/release/bin/kudu with pid 19174
I20260621 05:13:20.741319 19479 ts_tablet_manager.cc:1918] T c51ff510612f4d59ba97736d257d4296 P 245dfac09a1a4fcba9d4ed15c2387fc8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260621 05:13:20.741425 19479 log.cc:1196] T c51ff510612f4d59ba97736d257d4296 P 245dfac09a1a4fcba9d4ed15c2387fc8: Deleting WAL directory at /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782018616648578-3598-0/minicluster-data/ts-2/wal/wals/c51ff510612f4d59ba97736d257d4296
I20260621 05:13:20.741755 19479 ts_tablet_manager.cc:1939] T c51ff510612f4d59ba97736d257d4296 P 245dfac09a1a4fcba9d4ed15c2387fc8: Deleting consensus metadata
I20260621 05:13:20.742022 19451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:13:20.741859 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.131.129:41035 (local address 127.0.0.1:38352)
0621 05:13:20.741939 (+ 80us) negotiation.cc:107] Waiting for socket to connect
0621 05:13:20.741981 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.131.129:41035: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
I20260621 05:13:20.742264 18962 catalog_manager.cc:4841] TS 245dfac09a1a4fcba9d4ed15c2387fc8 (127.3.131.131:38877): tablet c51ff510612f4d59ba97736d257d4296 (table test-table [id=2dd763ba8d364a4b9eec4fe8548c9bd9]) successfully deleted
W20260621 05:13:20.742272 18965 catalog_manager.cc:4584] TS af09019aeb194a60b128a035658ed606 (127.3.131.129:41035): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet c51ff510612f4d59ba97736d257d4296: Network error: Client connection negotiation failed: client connection to 127.3.131.129:41035: connect: Connection refused (error 111)
I20260621 05:13:20.744117 19451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:13:20.743982 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.131.130:44723 (local address 127.0.0.1:56450)
0621 05:13:20.744041 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0621 05:13:20.744074 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.131.130:44723: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
I20260621 05:13:20.744203 3598 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWZOx2_/build/release/bin/kudu with pid 19308
I20260621 05:13:20.749571 3598 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWZOx2_/build/release/bin/kudu with pid 18946
I20260621 05:13:40.755963 3598 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWZOx2_/build/release/bin/kudu
/tmp/dist-test-taskWZOx2_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782018616648578-3598-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782018616648578-3598-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782018616648578-3598-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782018616648578-3598-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.131.190:46697
--webserver_interface=127.3.131.190
--webserver_port=35139
--keytab_file=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782018616648578-3598-0/krb5kdc/kudu_127.3.131.190.keytab
--principal=kudu/127.3.131.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.131.148:44007
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.131.190:46697
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782018616648578-3598-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782018616648578-3598-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782018616648578-3598-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 05:13:40.846252 19481 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 05:13:40.846493 19481 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 05:13:40.846517 19481 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 05:13:40.847810 19481 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260621 05:13:40.847865 19481 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260621 05:13:40.847878 19481 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260621 05:13:40.847891 19481 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 05:13:40.847904 19481 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 05:13:40.847918 19481 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 05:13:40.847929 19481 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 05:13:40.847985 19481 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 05:13:40.849606 19481 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.131.148:44007
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782018616648578-3598-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782018616648578-3598-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.3.131.190:46697
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782018616648578-3598-0/krb5kdc/kudu_127.3.131.190.keytab
--principal=kudu/127.3.131.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.131.190:46697
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782018616648578-3598-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.131.190
--webserver_port=35139
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19481
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782018616648578-3598-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 21 Jun 2026 04:16:16 UTC on ccycloud.quasar-xycaha.root.comops.site
build id 4457881
I20260621 05:13:40.849880 19481 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 05:13:40.850139 19481 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 05:13:40.859920 19488 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:13:40.859936 19490 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:13:40.859939 19487 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:13:40.860000 19481 server_base.cc:1034] running on GCE node
I20260621 05:13:40.860495 19481 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 05:13:40.860795 19481 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 05:13:40.861969 19481 hybrid_clock.cc:648] HybridClock initialized: now 1782018820861934 us; error 43 us; skew 500 ppm
Jun 21 05:13:40 dist-test-slave-vrq1 krb5kdc[18933](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782018820, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.131.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 05:13:40.869485 19481 init.cc:377] Logged in from keytab as kudu/127.3.131.190@KRBTEST.COM (short username kudu)
I20260621 05:13:40.869966 19481 webserver.cc:460] Webserver started at http://127.3.131.190:35139/ using document root <none> and password file <none>
I20260621 05:13:40.870178 19481 fs_manager.cc:359] Metadata directory not provided
I20260621 05:13:40.870234 19481 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 05:13:40.871876 19481 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260621 05:13:40.872668 19497 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 05:13:40.872926 19481 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260621 05:13:40.873021 19481 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782018616648578-3598-0/minicluster-data/master-0/data,/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782018616648578-3598-0/minicluster-data/master-0/wal
uuid: "5b3269e323af422bb256795282d54866"
format_stamp: "Formatted at 2026-06-21 05:13:20 on dist-test-slave-vrq1"
server_key: "456833c8468b3a016c85b28e4daff374"
server_key_iv: "0b2b5d432553b57c179b0a4863527276"
server_key_version: "encryptionkey@0"
I20260621 05:13:40.873435 19481 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782018616648578-3598-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782018616648578-3598-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782018616648578-3598-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 05:13:40.897548 19481 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 05:13:40.900317 19481 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 05:13:40.900462 19481 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 05:13:40.905349 19481 rpc_server.cc:307] RPC server started. Bound to: 127.3.131.190:46697
I20260621 05:13:40.905524 19549 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.131.190:46697 every 8 connection(s)
I20260621 05:13:40.905886 19481 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782018616648578-3598-0/minicluster-data/master-0/data/info.pb
I20260621 05:13:40.907804 19550 sys_catalog.cc:263] Verifying existing consensus state
I20260621 05:13:40.908685 19550 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5b3269e323af422bb256795282d54866: Bootstrap starting.
I20260621 05:13:40.912563 3598 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWZOx2_/build/release/bin/kudu as pid 19481
I20260621 05:13:40.912943 3598 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWZOx2_/build/release/bin/kudu
/tmp/dist-test-taskWZOx2_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782018616648578-3598-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782018616648578-3598-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782018616648578-3598-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782018616648578-3598-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.131.129:41035
--local_ip_for_outbound_sockets=127.3.131.129
--tserver_master_addrs=127.3.131.190:46697
--webserver_port=34091
--webserver_interface=127.3.131.129
--keytab_file=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782018616648578-3598-0/krb5kdc/kudu_127.3.131.129.keytab
--principal=kudu/127.3.131.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.131.148:44007
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782018616648578-3598-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782018616648578-3598-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782018616648578-3598-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260621 05:13:40.918886 19550 log.cc:824] T 00000000000000000000000000000000 P 5b3269e323af422bb256795282d54866: Log is configured to *not* fsync() on all Append() calls
I20260621 05:13:40.922803 19550 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5b3269e323af422bb256795282d54866: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260621 05:13:40.923327 19550 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5b3269e323af422bb256795282d54866: Bootstrap complete.
I20260621 05:13:40.927395 19550 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5b3269e323af422bb256795282d54866 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b3269e323af422bb256795282d54866" member_type: VOTER last_known_addr { host: "127.3.131.190" port: 46697 } }
I20260621 05:13:40.927958 19550 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5b3269e323af422bb256795282d54866 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b3269e323af422bb256795282d54866, State: Initialized, Role: FOLLOWER
I20260621 05:13:40.928194 19550 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5b3269e323af422bb256795282d54866 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b3269e323af422bb256795282d54866" member_type: VOTER last_known_addr { host: "127.3.131.190" port: 46697 } }
I20260621 05:13:40.928282 19550 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5b3269e323af422bb256795282d54866 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260621 05:13:40.928329 19550 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5b3269e323af422bb256795282d54866 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260621 05:13:40.928371 19550 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5b3269e323af422bb256795282d54866 [term 1 FOLLOWER]: Advancing to term 2
I20260621 05:13:40.929847 19550 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5b3269e323af422bb256795282d54866 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b3269e323af422bb256795282d54866" member_type: VOTER last_known_addr { host: "127.3.131.190" port: 46697 } }
I20260621 05:13:40.929970 19550 leader_election.cc:302] T 00000000000000000000000000000000 P 5b3269e323af422bb256795282d54866 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5b3269e323af422bb256795282d54866; no voters:
I20260621 05:13:40.930160 19550 leader_election.cc:288] T 00000000000000000000000000000000 P 5b3269e323af422bb256795282d54866 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260621 05:13:40.930284 19554 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5b3269e323af422bb256795282d54866 [term 2 FOLLOWER]: Leader election won for term 2
I20260621 05:13:40.930439 19550 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5b3269e323af422bb256795282d54866 [sys.catalog]: configured and running, proceeding with master startup.
I20260621 05:13:40.930522 19554 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5b3269e323af422bb256795282d54866 [term 2 LEADER]: Becoming Leader. State: Replica: 5b3269e323af422bb256795282d54866, State: Running, Role: LEADER
I20260621 05:13:40.930626 19554 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5b3269e323af422bb256795282d54866 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b3269e323af422bb256795282d54866" member_type: VOTER last_known_addr { host: "127.3.131.190" port: 46697 } }
I20260621 05:13:40.930984 19555 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5b3269e323af422bb256795282d54866 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "5b3269e323af422bb256795282d54866" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b3269e323af422bb256795282d54866" member_type: VOTER last_known_addr { host: "127.3.131.190" port: 46697 } } }
I20260621 05:13:40.931077 19555 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5b3269e323af422bb256795282d54866 [sys.catalog]: This master's current role is: LEADER
I20260621 05:13:40.931355 19556 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5b3269e323af422bb256795282d54866 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5b3269e323af422bb256795282d54866. Latest consensus state: current_term: 2 leader_uuid: "5b3269e323af422bb256795282d54866" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b3269e323af422bb256795282d54866" member_type: VOTER last_known_addr { host: "127.3.131.190" port: 46697 } } }
I20260621 05:13:40.931427 19556 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5b3269e323af422bb256795282d54866 [sys.catalog]: This master's current role is: LEADER
I20260621 05:13:40.931536 19563 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260621 05:13:40.932416 19563 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260621 05:13:40.932526 19563 catalog_manager.cc:1259] Loaded cluster ID: dce3d0e84ae94f97814f61bb245e0d57
I20260621 05:13:40.932657 19563 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260621 05:13:40.933992 19563 catalog_manager.cc:1504] Loading token signing keys...
I20260621 05:13:40.934183 19563 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 5b3269e323af422bb256795282d54866: Loaded TSK: 0
I20260621 05:13:40.943642 19563 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5b3269e323af422bb256795282d54866: Generated new TSK 1
I20260621 05:13:40.943774 19563 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260621 05:13:41.006158 19552 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 05:13:41.006448 19552 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 05:13:41.006495 19552 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 05:13:41.007966 19552 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 05:13:41.008049 19552 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 05:13:41.008101 19552 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.131.129
I20260621 05:13:41.009860 19552 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.131.148:44007
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782018616648578-3598-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782018616648578-3598-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782018616648578-3598-0/krb5kdc/kudu_127.3.131.129.keytab
--principal=kudu/127.3.131.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.131.129:41035
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782018616648578-3598-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.131.129
--webserver_port=34091
--tserver_master_addrs=127.3.131.190:46697
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19552
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.131.129
--log_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782018616648578-3598-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 21 Jun 2026 04:16:16 UTC on ccycloud.quasar-xycaha.root.comops.site
build id 4457881
I20260621 05:13:41.010157 19552 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 05:13:41.010412 19552 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 05:13:41.016130 19577 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:13:41.016085 19580 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:13:41.016121 19578 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:13:41.017165 19552 server_base.cc:1034] running on GCE node
I20260621 05:13:41.017442 19552 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 05:13:41.017786 19552 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 05:13:41.019119 19552 hybrid_clock.cc:648] HybridClock initialized: now 1782018821019087 us; error 42 us; skew 500 ppm
Jun 21 05:13:41 dist-test-slave-vrq1 krb5kdc[18933](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782018821, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.131.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 05:13:41.022578 19552 init.cc:377] Logged in from keytab as kudu/127.3.131.129@KRBTEST.COM (short username kudu)
I20260621 05:13:41.023216 19552 webserver.cc:460] Webserver started at http://127.3.131.129:34091/ using document root <none> and password file <none>
I20260621 05:13:41.023483 19552 fs_manager.cc:359] Metadata directory not provided
I20260621 05:13:41.023548 19552 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 05:13:41.025127 19552 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260621 05:13:41.026371 19587 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 05:13:41.026631 19552 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260621 05:13:41.026746 19552 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782018616648578-3598-0/minicluster-data/ts-0/data,/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782018616648578-3598-0/minicluster-data/ts-0/wal
uuid: "af09019aeb194a60b128a035658ed606"
format_stamp: "Formatted at 2026-06-21 05:13:20 on dist-test-slave-vrq1"
server_key: "3a7917f757ceb859a74758824cce2404"
server_key_iv: "6d67965e121b33fd4eed38e9a363377a"
server_key_version: "encryptionkey@0"
I20260621 05:13:41.027153 19552 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782018616648578-3598-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782018616648578-3598-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782018616648578-3598-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 05:13:41.036067 19552 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 05:13:41.038443 19552 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 05:13:41.038625 19552 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 05:13:41.038867 19552 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 05:13:41.039361 19594 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260621 05:13:41.041285 19552 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260621 05:13:41.041394 19552 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260621 05:13:41.041446 19552 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260621 05:13:41.042094 19552 ts_tablet_manager.cc:610] Registered 1 tablets
I20260621 05:13:41.042148 19552 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260621 05:13:41.042222 19594 tablet_bootstrap.cc:492] T c51ff510612f4d59ba97736d257d4296 P af09019aeb194a60b128a035658ed606: Bootstrap starting.
I20260621 05:13:41.049495 19552 rpc_server.cc:307] RPC server started. Bound to: 127.3.131.129:41035
I20260621 05:13:41.049914 19552 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782018616648578-3598-0/minicluster-data/ts-0/data/info.pb
I20260621 05:13:41.053416 19594 log.cc:824] T c51ff510612f4d59ba97736d257d4296 P af09019aeb194a60b128a035658ed606: Log is configured to *not* fsync() on all Append() calls
I20260621 05:13:41.053839 19701 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.131.129:41035 every 8 connection(s)
I20260621 05:13:41.056391 19594 tablet_bootstrap.cc:492] T c51ff510612f4d59ba97736d257d4296 P af09019aeb194a60b128a035658ed606: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260621 05:13:41.056751 19594 tablet_bootstrap.cc:492] T c51ff510612f4d59ba97736d257d4296 P af09019aeb194a60b128a035658ed606: Bootstrap complete.
I20260621 05:13:41.057973 19594 ts_tablet_manager.cc:1397] T c51ff510612f4d59ba97736d257d4296 P af09019aeb194a60b128a035658ed606: Time spent bootstrapping tablet: real 0.016s user 0.004s sys 0.009s
Jun 21 05:13:41 dist-test-slave-vrq1 krb5kdc[18933](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782018821, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.131.129@KRBTEST.COM for kudu/127.3.131.190@KRBTEST.COM
I20260621 05:13:41.059346 19594 raft_consensus.cc:348] T c51ff510612f4d59ba97736d257d4296 P af09019aeb194a60b128a035658ed606 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "245dfac09a1a4fcba9d4ed15c2387fc8" member_type: VOTER last_known_addr { host: "127.3.131.131" port: 38877 } } peers { permanent_uuid: "518697964a27412aa91818fcc615e6d5" member_type: VOTER last_known_addr { host: "127.3.131.130" port: 44723 } } peers { permanent_uuid: "af09019aeb194a60b128a035658ed606" member_type: VOTER last_known_addr { host: "127.3.131.129" port: 41035 } }
I20260621 05:13:41.059701 19594 raft_consensus.cc:729] T c51ff510612f4d59ba97736d257d4296 P af09019aeb194a60b128a035658ed606 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: af09019aeb194a60b128a035658ed606, State: Initialized, Role: FOLLOWER
I20260621 05:13:41.059873 19594 consensus_queue.cc:260] T c51ff510612f4d59ba97736d257d4296 P af09019aeb194a60b128a035658ed606 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "245dfac09a1a4fcba9d4ed15c2387fc8" member_type: VOTER last_known_addr { host: "127.3.131.131" port: 38877 } } peers { permanent_uuid: "518697964a27412aa91818fcc615e6d5" member_type: VOTER last_known_addr { host: "127.3.131.130" port: 44723 } } peers { permanent_uuid: "af09019aeb194a60b128a035658ed606" member_type: VOTER last_known_addr { host: "127.3.131.129" port: 41035 } }
I20260621 05:13:41.060003 3598 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWZOx2_/build/release/bin/kudu as pid 19552
I20260621 05:13:41.060178 19594 ts_tablet_manager.cc:1428] T c51ff510612f4d59ba97736d257d4296 P af09019aeb194a60b128a035658ed606: Time spent starting tablet: real 0.002s user 0.004s sys 0.000s
I20260621 05:13:41.060293 3598 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWZOx2_/build/release/bin/kudu
/tmp/dist-test-taskWZOx2_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782018616648578-3598-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782018616648578-3598-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782018616648578-3598-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782018616648578-3598-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.131.130:44723
--local_ip_for_outbound_sockets=127.3.131.130
--tserver_master_addrs=127.3.131.190:46697
--webserver_port=34509
--webserver_interface=127.3.131.130
--keytab_file=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782018616648578-3598-0/krb5kdc/kudu_127.3.131.130.keytab
--principal=kudu/127.3.131.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.131.148:44007
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782018616648578-3598-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782018616648578-3598-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782018616648578-3598-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260621 05:13:41.066422 19704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:13:41.050620 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.131.190:46697 (local address 127.3.131.129:48729)
0621 05:13:41.050815 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0621 05:13:41.050858 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0621 05:13:41.051476 (+ 618us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:13:41.054094 (+ 2618us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:13:41.054099 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:13:41.054379 (+ 280us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:13:41.054570 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:13:41.054575 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:13:41.055987 (+ 1412us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:13:41.055988 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:13:41.056925 (+ 937us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:13:41.056928 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:13:41.057534 (+ 606us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:13:41.057535 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:13:41.057675 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:13:41.058231 (+ 556us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 05:13:41.058239 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:13:41.059724 (+ 1485us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 05:13:41.064286 (+ 4562us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:13:41.064298 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:13:41.064301 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:13:41.064666 (+ 365us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:13:41.065017 (+ 351us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:13:41.065018 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:13:41.065018 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:13:41.065167 (+ 149us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:13:41.065700 (+ 533us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 05:13:41.065707 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 05:13:41.065875 (+ 168us) client_negotiation.cc:769] Sending connection context
0621 05:13:41.065992 (+ 117us) client_negotiation.cc:240] Negotiation successful
0621 05:13:41.066182 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":116,"threads_started":1}
I20260621 05:13:41.066913 19705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:13:41.050732 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.131.129:48729 (local address 127.3.131.190:46697)
0621 05:13:41.053558 (+ 2826us) server_negotiation.cc:206] Beginning negotiation
0621 05:13:41.053569 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0621 05:13:41.053612 (+ 43us) server_negotiation.cc:407] Connection header received
0621 05:13:41.053761 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:13:41.053769 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:13:41.053888 (+ 119us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:13:41.054679 (+ 791us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:13:41.054743 (+ 64us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:13:41.055866 (+ 1123us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:13:41.057019 (+ 1153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:13:41.057434 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:13:41.057560 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:13:41.061699 (+ 4139us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:13:41.061706 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:13:41.061706 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:13:41.061722 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:13:41.064163 (+ 2441us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:13:41.064772 (+ 609us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:13:41.064777 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:13:41.064777 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:13:41.064929 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:13:41.065246 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:13:41.065247 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:13:41.065248 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:13:41.065613 (+ 365us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:13:41.065712 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0621 05:13:41.066081 (+ 369us) server_negotiation.cc:299] Negotiation successful
0621 05:13:41.066250 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2762,"thread_start_us":85,"threads_started":1}
I20260621 05:13:41.067694 19702 heartbeater.cc:347] Connected to a master server at 127.3.131.190:46697
I20260621 05:13:41.067816 19702 heartbeater.cc:464] Registering TS with master...
I20260621 05:13:41.068049 19702 heartbeater.cc:511] Master 127.3.131.190:46697 requested a full tablet report, sending...
I20260621 05:13:41.068943 19514 ts_manager.cc:194] Registered new tserver with Master: af09019aeb194a60b128a035658ed606 (127.3.131.129:41035)
I20260621 05:13:41.070305 19514 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.131.129@KRBTEST.COM'} at 127.3.131.129:48729
I20260621 05:13:41.071458 19702 heartbeater.cc:503] Master 127.3.131.190:46697 was elected leader, sending a full tablet report...
Jun 21 05:13:41 dist-test-slave-vrq1 krb5kdc[18933](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782018820, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.131.190@KRBTEST.COM for kudu/127.3.131.129@KRBTEST.COM
I20260621 05:13:41.080902 19709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:13:41.069723 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.131.129:41035 (local address 127.0.0.1:60660)
0621 05:13:41.070143 (+ 420us) negotiation.cc:107] Waiting for socket to connect
0621 05:13:41.070180 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0621 05:13:41.070248 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:13:41.070787 (+ 539us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:13:41.070792 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:13:41.071245 (+ 453us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:13:41.071328 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:13:41.071332 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:13:41.072821 (+ 1489us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:13:41.072822 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:13:41.073761 (+ 939us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:13:41.073765 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:13:41.074388 (+ 623us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:13:41.074389 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:13:41.074469 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:13:41.075115 (+ 646us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 05:13:41.075121 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:13:41.076936 (+ 1815us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 05:13:41.079465 (+ 2529us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:13:41.079474 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:13:41.079477 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:13:41.079771 (+ 294us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:13:41.080023 (+ 252us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:13:41.080023 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:13:41.080024 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:13:41.080099 (+ 75us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:13:41.080454 (+ 355us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 05:13:41.080462 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 05:13:41.080527 (+ 65us) client_negotiation.cc:769] Sending connection context
0621 05:13:41.080641 (+ 114us) client_negotiation.cc:240] Negotiation successful
0621 05:13:41.080784 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":390,"thread_start_us":102,"threads_started":1}
I20260621 05:13:41.080904 19710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:13:41.069990 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60660 (local address 127.3.131.129:41035)
0621 05:13:41.070370 (+ 380us) server_negotiation.cc:206] Beginning negotiation
0621 05:13:41.070375 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 05:13:41.070383 (+ 8us) server_negotiation.cc:407] Connection header received
0621 05:13:41.070426 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:13:41.070429 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:13:41.070445 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:13:41.070513 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:13:41.071440 (+ 927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:13:41.072725 (+ 1285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:13:41.073857 (+ 1132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:13:41.074304 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:13:41.074377 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:13:41.077050 (+ 2673us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:13:41.077060 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:13:41.077061 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:13:41.077070 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:13:41.079347 (+ 2277us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:13:41.079868 (+ 521us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:13:41.079872 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:13:41.079872 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:13:41.079943 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:13:41.080169 (+ 226us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:13:41.080170 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:13:41.080170 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:13:41.080374 (+ 204us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:13:41.080458 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0621 05:13:41.080695 (+ 237us) server_negotiation.cc:299] Negotiation successful
0621 05:13:41.080801 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":351,"thread_start_us":71,"threads_started":1}
I20260621 05:13:41.081429 19622 tablet_service.cc:1514] Processing DeleteTablet for tablet c51ff510612f4d59ba97736d257d4296 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 05:13:20 UTC) from {username='kudu', principal='kudu/127.3.131.190@KRBTEST.COM'} at 127.0.0.1:60660
I20260621 05:13:41.083313 19711 tablet_replica.cc:330] stopping tablet replica
I20260621 05:13:41.083441 19711 raft_consensus.cc:2229] T c51ff510612f4d59ba97736d257d4296 P af09019aeb194a60b128a035658ed606 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260621 05:13:41.083510 19711 raft_consensus.cc:2258] T c51ff510612f4d59ba97736d257d4296 P af09019aeb194a60b128a035658ed606 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 05:13:41.085491 19711 ts_tablet_manager.cc:1905] T c51ff510612f4d59ba97736d257d4296 P af09019aeb194a60b128a035658ed606: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 05:13:41.087785 19711 ts_tablet_manager.cc:1918] T c51ff510612f4d59ba97736d257d4296 P af09019aeb194a60b128a035658ed606: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260621 05:13:41.087867 19711 log.cc:1196] T c51ff510612f4d59ba97736d257d4296 P af09019aeb194a60b128a035658ed606: Deleting WAL directory at /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782018616648578-3598-0/minicluster-data/ts-0/wal/wals/c51ff510612f4d59ba97736d257d4296
I20260621 05:13:41.088086 19711 ts_tablet_manager.cc:1939] T c51ff510612f4d59ba97736d257d4296 P af09019aeb194a60b128a035658ed606: Deleting consensus metadata
I20260621 05:13:41.089004 19501 catalog_manager.cc:4841] TS af09019aeb194a60b128a035658ed606 (127.3.131.129:41035): tablet c51ff510612f4d59ba97736d257d4296 (table test-table [id=2dd763ba8d364a4b9eec4fe8548c9bd9]) successfully deleted
W20260621 05:13:41.152725 19708 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 05:13:41.152944 19708 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 05:13:41.152969 19708 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 05:13:41.154245 19708 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 05:13:41.154296 19708 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 05:13:41.154326 19708 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.131.130
I20260621 05:13:41.155804 19708 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.131.148:44007
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782018616648578-3598-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782018616648578-3598-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782018616648578-3598-0/krb5kdc/kudu_127.3.131.130.keytab
--principal=kudu/127.3.131.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.131.130:44723
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782018616648578-3598-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.131.130
--webserver_port=34509
--tserver_master_addrs=127.3.131.190:46697
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19708
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.131.130
--log_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782018616648578-3598-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 21 Jun 2026 04:16:16 UTC on ccycloud.quasar-xycaha.root.comops.site
build id 4457881
I20260621 05:13:41.156068 19708 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 05:13:41.156286 19708 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 05:13:41.161602 19717 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:13:41.161624 19719 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:13:41.161676 19716 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:13:41.161830 19708 server_base.cc:1034] running on GCE node
I20260621 05:13:41.162130 19708 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 05:13:41.162395 19708 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 05:13:41.163555 19708 hybrid_clock.cc:648] HybridClock initialized: now 1782018821163540 us; error 30 us; skew 500 ppm
Jun 21 05:13:41 dist-test-slave-vrq1 krb5kdc[18933](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782018821, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.131.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 05:13:41.166400 19708 init.cc:377] Logged in from keytab as kudu/127.3.131.130@KRBTEST.COM (short username kudu)
I20260621 05:13:41.166949 19708 webserver.cc:460] Webserver started at http://127.3.131.130:34509/ using document root <none> and password file <none>
I20260621 05:13:41.167280 19708 fs_manager.cc:359] Metadata directory not provided
I20260621 05:13:41.167347 19708 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 05:13:41.168928 19708 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260621 05:13:41.169756 19726 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 05:13:41.169968 19708 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260621 05:13:41.170082 19708 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782018616648578-3598-0/minicluster-data/ts-1/data,/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782018616648578-3598-0/minicluster-data/ts-1/wal
uuid: "518697964a27412aa91818fcc615e6d5"
format_stamp: "Formatted at 2026-06-21 05:13:20 on dist-test-slave-vrq1"
server_key: "6a04f3ab56b30c3811a656e5d322d68f"
server_key_iv: "cf756f1d79126e3538eab75edec77821"
server_key_version: "encryptionkey@0"
I20260621 05:13:41.170447 19708 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782018616648578-3598-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782018616648578-3598-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782018616648578-3598-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 05:13:41.179824 19708 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 05:13:41.182346 19708 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 05:13:41.182529 19708 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 05:13:41.182937 19708 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 05:13:41.183534 19733 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
W20260621 05:13:41.185407 19733 ts_tablet_manager.cc:1852] T c51ff510612f4d59ba97736d257d4296 P 518697964a27412aa91818fcc615e6d5: Tablet Manager startup: Rolling forward tablet deletion of type TABLET_DATA_DELETED
I20260621 05:13:41.185846 19733 ts_tablet_manager.cc:1905] T c51ff510612f4d59ba97736d257d4296 P 518697964a27412aa91818fcc615e6d5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 05:13:41.188199 19733 ts_tablet_manager.cc:1918] T c51ff510612f4d59ba97736d257d4296 P 518697964a27412aa91818fcc615e6d5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260621 05:13:41.188303 19733 log.cc:1196] T c51ff510612f4d59ba97736d257d4296 P 518697964a27412aa91818fcc615e6d5: Deleting WAL directory at /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782018616648578-3598-0/minicluster-data/ts-1/wal/wals/c51ff510612f4d59ba97736d257d4296
I20260621 05:13:41.188591 19733 ts_tablet_manager.cc:1939] T c51ff510612f4d59ba97736d257d4296 P 518697964a27412aa91818fcc615e6d5: Deleting consensus metadata
I20260621 05:13:41.188884 19708 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 0 live tablets)
I20260621 05:13:41.188973 19708 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.006s user 0.000s sys 0.000s
I20260621 05:13:41.189023 19708 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 05:13:41.189055 19708 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 05:13:41.195978 19708 rpc_server.cc:307] RPC server started. Bound to: 127.3.131.130:44723
I20260621 05:13:41.196071 19840 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.131.130:44723 every 8 connection(s)
I20260621 05:13:41.196434 19708 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782018616648578-3598-0/minicluster-data/ts-1/data/info.pb
I20260621 05:13:41.196728 3598 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWZOx2_/build/release/bin/kudu as pid 19708
I20260621 05:13:41.197050 3598 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWZOx2_/build/release/bin/kudu
/tmp/dist-test-taskWZOx2_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782018616648578-3598-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782018616648578-3598-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782018616648578-3598-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782018616648578-3598-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.131.131:38877
--local_ip_for_outbound_sockets=127.3.131.131
--tserver_master_addrs=127.3.131.190:46697
--webserver_port=43527
--webserver_interface=127.3.131.131
--keytab_file=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782018616648578-3598-0/krb5kdc/kudu_127.3.131.131.keytab
--principal=kudu/127.3.131.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.131.148:44007
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782018616648578-3598-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782018616648578-3598-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782018616648578-3598-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 21 05:13:41 dist-test-slave-vrq1 krb5kdc[18933](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782018821, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.131.130@KRBTEST.COM for kudu/127.3.131.190@KRBTEST.COM
I20260621 05:13:41.206892 19705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:13:41.197297 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.131.130:38433 (local address 127.3.131.190:46697)
0621 05:13:41.197387 (+ 90us) server_negotiation.cc:206] Beginning negotiation
0621 05:13:41.197389 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:13:41.197629 (+ 240us) server_negotiation.cc:407] Connection header received
0621 05:13:41.198268 (+ 639us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:13:41.198269 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:13:41.198287 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:13:41.198371 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:13:41.199097 (+ 726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:13:41.199746 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:13:41.200887 (+ 1141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:13:41.201127 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:13:41.201220 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:13:41.203699 (+ 2479us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:13:41.203701 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:13:41.203701 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:13:41.203710 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:13:41.205293 (+ 1583us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:13:41.205813 (+ 520us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:13:41.205814 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:13:41.205814 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:13:41.205882 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:13:41.206169 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:13:41.206170 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:13:41.206171 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:13:41.206305 (+ 134us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:13:41.206381 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0621 05:13:41.206684 (+ 303us) server_negotiation.cc:299] Negotiation successful
0621 05:13:41.206797 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260621 05:13:41.207152 19843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:13:41.197289 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.131.190:46697 (local address 127.3.131.130:38433)
0621 05:13:41.197486 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0621 05:13:41.197527 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0621 05:13:41.198142 (+ 615us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:13:41.198460 (+ 318us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:13:41.198464 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:13:41.198764 (+ 300us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:13:41.198982 (+ 218us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:13:41.198987 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:13:41.199892 (+ 905us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:13:41.199898 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:13:41.200783 (+ 885us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:13:41.200786 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:13:41.201240 (+ 454us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:13:41.201241 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:13:41.201385 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:13:41.202031 (+ 646us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 05:13:41.202040 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:13:41.203583 (+ 1543us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 05:13:41.205422 (+ 1839us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:13:41.205431 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:13:41.205434 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:13:41.205715 (+ 281us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:13:41.205965 (+ 250us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:13:41.205966 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:13:41.205966 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:13:41.206088 (+ 122us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:13:41.206381 (+ 293us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 05:13:41.206389 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 05:13:41.206545 (+ 156us) client_negotiation.cc:769] Sending connection context
0621 05:13:41.206661 (+ 116us) client_negotiation.cc:240] Negotiation successful
0621 05:13:41.206835 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":103,"threads_started":1}
I20260621 05:13:41.207756 19841 heartbeater.cc:347] Connected to a master server at 127.3.131.190:46697
I20260621 05:13:41.207880 19841 heartbeater.cc:464] Registering TS with master...
I20260621 05:13:41.208073 19841 heartbeater.cc:511] Master 127.3.131.190:46697 requested a full tablet report, sending...
I20260621 05:13:41.208487 19514 ts_manager.cc:194] Registered new tserver with Master: 518697964a27412aa91818fcc615e6d5 (127.3.131.130:44723)
I20260621 05:13:41.208913 19514 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.131.130@KRBTEST.COM'} at 127.3.131.130:38433
W20260621 05:13:41.287101 19844 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 05:13:41.287410 19844 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 05:13:41.287438 19844 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 05:13:41.288709 19844 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 05:13:41.288769 19844 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 05:13:41.288798 19844 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.131.131
I20260621 05:13:41.290373 19844 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.131.148:44007
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782018616648578-3598-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782018616648578-3598-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782018616648578-3598-0/krb5kdc/kudu_127.3.131.131.keytab
--principal=kudu/127.3.131.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.131.131:38877
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782018616648578-3598-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.131.131
--webserver_port=43527
--tserver_master_addrs=127.3.131.190:46697
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19844
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.131.131
--log_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782018616648578-3598-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 21 Jun 2026 04:16:16 UTC on ccycloud.quasar-xycaha.root.comops.site
build id 4457881
I20260621 05:13:41.290684 19844 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 05:13:41.290941 19844 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 05:13:41.296066 19849 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:13:41.302438 19850 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:13:41.303073 19852 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:13:41.303108 19844 server_base.cc:1034] running on GCE node
I20260621 05:13:41.303457 19844 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 05:13:41.303802 19844 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 05:13:41.304956 19844 hybrid_clock.cc:648] HybridClock initialized: now 1782018821304932 us; error 39 us; skew 500 ppm
Jun 21 05:13:41 dist-test-slave-vrq1 krb5kdc[18933](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782018821, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.131.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 05:13:41.308408 19844 init.cc:377] Logged in from keytab as kudu/127.3.131.131@KRBTEST.COM (short username kudu)
I20260621 05:13:41.308938 19844 webserver.cc:460] Webserver started at http://127.3.131.131:43527/ using document root <none> and password file <none>
I20260621 05:13:41.309195 19844 fs_manager.cc:359] Metadata directory not provided
I20260621 05:13:41.309288 19844 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 05:13:41.310885 19844 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260621 05:13:41.311805 19859 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 05:13:41.312031 19844 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260621 05:13:41.312141 19844 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782018616648578-3598-0/minicluster-data/ts-2/data,/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782018616648578-3598-0/minicluster-data/ts-2/wal
uuid: "245dfac09a1a4fcba9d4ed15c2387fc8"
format_stamp: "Formatted at 2026-06-21 05:13:20 on dist-test-slave-vrq1"
server_key: "067ea0ddafaa5328ebc1fb466104a9e8"
server_key_iv: "417bfc56a463306fd6117f4085658fee"
server_key_version: "encryptionkey@0"
I20260621 05:13:41.312542 19844 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782018616648578-3598-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782018616648578-3598-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782018616648578-3598-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 05:13:41.323957 19844 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 05:13:41.326327 19844 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 05:13:41.326511 19844 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 05:13:41.326772 19844 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 05:13:41.327098 19844 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 05:13:41.327155 19844 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 05:13:41.327215 19844 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 05:13:41.327277 19844 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 05:13:41.334244 19844 rpc_server.cc:307] RPC server started. Bound to: 127.3.131.131:38877
I20260621 05:13:41.334281 19972 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.131.131:38877 every 8 connection(s)
I20260621 05:13:41.334676 19844 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782018616648578-3598-0/minicluster-data/ts-2/data/info.pb
Jun 21 05:13:41 dist-test-slave-vrq1 krb5kdc[18933](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782018821, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.131.131@KRBTEST.COM for kudu/127.3.131.190@KRBTEST.COM
I20260621 05:13:41.343434 3598 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWZOx2_/build/release/bin/kudu as pid 19844
I20260621 05:13:41.345100 19705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:13:41.335604 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.131.131:46907 (local address 127.3.131.190:46697)
0621 05:13:41.335667 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0621 05:13:41.335669 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:13:41.335921 (+ 252us) server_negotiation.cc:407] Connection header received
0621 05:13:41.336457 (+ 536us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:13:41.336458 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:13:41.336471 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:13:41.336517 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:13:41.337505 (+ 988us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:13:41.338017 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:13:41.339048 (+ 1031us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:13:41.339336 (+ 288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:13:41.339427 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:13:41.341748 (+ 2321us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:13:41.341750 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:13:41.341750 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:13:41.341759 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:13:41.343295 (+ 1536us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:13:41.343973 (+ 678us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:13:41.343975 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:13:41.343976 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:13:41.344049 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:13:41.344383 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:13:41.344384 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:13:41.344384 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:13:41.344517 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:13:41.344593 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0621 05:13:41.344912 (+ 319us) server_negotiation.cc:299] Negotiation successful
0621 05:13:41.345006 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260621 05:13:41.345337 19975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:13:41.335579 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.131.190:46697 (local address 127.3.131.131:46907)
0621 05:13:41.335798 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0621 05:13:41.335835 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0621 05:13:41.336347 (+ 512us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:13:41.336588 (+ 241us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:13:41.336592 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:13:41.337192 (+ 600us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:13:41.337398 (+ 206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:13:41.337403 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:13:41.338120 (+ 717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:13:41.338121 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:13:41.338947 (+ 826us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:13:41.338950 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:13:41.339440 (+ 490us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:13:41.339441 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:13:41.339573 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:13:41.340085 (+ 512us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 05:13:41.340094 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:13:41.341613 (+ 1519us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 05:13:41.343398 (+ 1785us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:13:41.343408 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:13:41.343411 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:13:41.343833 (+ 422us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:13:41.344143 (+ 310us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:13:41.344144 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:13:41.344144 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:13:41.344297 (+ 153us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:13:41.344595 (+ 298us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 05:13:41.344602 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 05:13:41.344752 (+ 150us) client_negotiation.cc:769] Sending connection context
0621 05:13:41.344880 (+ 128us) client_negotiation.cc:240] Negotiation successful
0621 05:13:41.345064 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":88,"threads_started":1}
I20260621 05:13:41.346051 19973 heartbeater.cc:347] Connected to a master server at 127.3.131.190:46697
I20260621 05:13:41.346170 19973 heartbeater.cc:464] Registering TS with master...
I20260621 05:13:41.346423 19973 heartbeater.cc:511] Master 127.3.131.190:46697 requested a full tablet report, sending...
I20260621 05:13:41.346913 19514 ts_manager.cc:194] Registered new tserver with Master: 245dfac09a1a4fcba9d4ed15c2387fc8 (127.3.131.131:38877)
I20260621 05:13:41.347424 19514 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.131.131@KRBTEST.COM'} at 127.3.131.131:46907
I20260621 05:13:41.351328 19977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:13:41.343946 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41460 (local address 127.3.131.190:46697)
0621 05:13:41.344068 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0621 05:13:41.344070 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:13:41.344092 (+ 22us) server_negotiation.cc:407] Connection header received
0621 05:13:41.344164 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:13:41.344165 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:13:41.344177 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:13:41.344233 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:13:41.344837 (+ 604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:13:41.345570 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:13:41.346770 (+ 1200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:13:41.347079 (+ 309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:13:41.347148 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:13:41.348578 (+ 1430us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:13:41.348579 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:13:41.348580 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:13:41.348586 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:13:41.349989 (+ 1403us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:13:41.350353 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:13:41.350354 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:13:41.350355 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:13:41.350409 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:13:41.350677 (+ 268us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:13:41.350678 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:13:41.350679 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:13:41.350845 (+ 166us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:13:41.350995 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0621 05:13:41.351064 (+ 69us) server_negotiation.cc:299] Negotiation successful
0621 05:13:41.351137 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":96,"thread_start_us":66,"threads_started":1}
I20260621 05:13:41.352299 3598 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260621 05:13:41.355383 19977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:13:41.352657 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41474 (local address 127.3.131.190:46697)
0621 05:13:41.352720 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0621 05:13:41.352722 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:13:41.352854 (+ 132us) server_negotiation.cc:407] Connection header received
0621 05:13:41.352902 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:13:41.352902 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:13:41.352935 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:13:41.352989 (+ 54us) server_negotiation.cc:226] Negotiated authn=TOKEN
0621 05:13:41.353068 (+ 79us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:13:41.353556 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:13:41.354533 (+ 977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:13:41.354753 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:13:41.354818 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:13:41.354984 (+ 166us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0621 05:13:41.355033 (+ 49us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0621 05:13:41.355313 (+ 280us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:41474: token expired
Metrics: {"server-negotiator.queue_time_us":40}
W20260621 05:13:41.355409 19441 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.3.131.190:46697)
W20260621 05:13:41.355461 19977 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:41474: token expired
I20260621 05:13:41.362138 19977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:13:41.355693 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41480 (local address 127.3.131.190:46697)
0621 05:13:41.355730 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0621 05:13:41.355731 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0621 05:13:41.355741 (+ 10us) server_negotiation.cc:407] Connection header received
0621 05:13:41.355809 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:13:41.355810 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:13:41.355817 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:13:41.355863 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:13:41.356365 (+ 502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:13:41.356865 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:13:41.357660 (+ 795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:13:41.357925 (+ 265us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:13:41.357987 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:13:41.359485 (+ 1498us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:13:41.359487 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:13:41.359487 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:13:41.359493 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:13:41.361100 (+ 1607us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:13:41.361448 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:13:41.361449 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:13:41.361449 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:13:41.361499 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:13:41.361723 (+ 224us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:13:41.361724 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:13:41.361724 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:13:41.361822 (+ 98us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:13:41.361896 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0621 05:13:41.361980 (+ 84us) server_negotiation.cc:299] Negotiation successful
0621 05:13:41.362041 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260621 05:13:41.363469 19514 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41480:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260621 05:13:41.365123 19514 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260621 05:13:41.367513 19622 tablet_service.cc:1467] Processing CreateTablet for tablet 6a2f678342444e57bb2bccae6c6af0c9 (DEFAULT_TABLE table=test-table [id=5034638cf3c343a5a47465e07054b90f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 05:13:41.367760 19622 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6a2f678342444e57bb2bccae6c6af0c9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 05:13:41.369408 19594 tablet_bootstrap.cc:492] T 6a2f678342444e57bb2bccae6c6af0c9 P af09019aeb194a60b128a035658ed606: Bootstrap starting.
I20260621 05:13:41.369721 19594 tablet_bootstrap.cc:654] T 6a2f678342444e57bb2bccae6c6af0c9 P af09019aeb194a60b128a035658ed606: Neither blocks nor log segments found. Creating new log.
I20260621 05:13:41.370592 19594 tablet_bootstrap.cc:492] T 6a2f678342444e57bb2bccae6c6af0c9 P af09019aeb194a60b128a035658ed606: No bootstrap required, opened a new log
I20260621 05:13:41.370652 19594 ts_tablet_manager.cc:1397] T 6a2f678342444e57bb2bccae6c6af0c9 P af09019aeb194a60b128a035658ed606: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260621 05:13:41.370852 19594 raft_consensus.cc:348] T 6a2f678342444e57bb2bccae6c6af0c9 P af09019aeb194a60b128a035658ed606 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "518697964a27412aa91818fcc615e6d5" member_type: VOTER last_known_addr { host: "127.3.131.130" port: 44723 } } peers { permanent_uuid: "245dfac09a1a4fcba9d4ed15c2387fc8" member_type: VOTER last_known_addr { host: "127.3.131.131" port: 38877 } } peers { permanent_uuid: "af09019aeb194a60b128a035658ed606" member_type: VOTER last_known_addr { host: "127.3.131.129" port: 41035 } }
I20260621 05:13:41.370918 19594 raft_consensus.cc:374] T 6a2f678342444e57bb2bccae6c6af0c9 P af09019aeb194a60b128a035658ed606 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 05:13:41.370951 19594 raft_consensus.cc:729] T 6a2f678342444e57bb2bccae6c6af0c9 P af09019aeb194a60b128a035658ed606 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af09019aeb194a60b128a035658ed606, State: Initialized, Role: FOLLOWER
I20260621 05:13:41.371009 19594 consensus_queue.cc:260] T 6a2f678342444e57bb2bccae6c6af0c9 P af09019aeb194a60b128a035658ed606 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "518697964a27412aa91818fcc615e6d5" member_type: VOTER last_known_addr { host: "127.3.131.130" port: 44723 } } peers { permanent_uuid: "245dfac09a1a4fcba9d4ed15c2387fc8" member_type: VOTER last_known_addr { host: "127.3.131.131" port: 38877 } } peers { permanent_uuid: "af09019aeb194a60b128a035658ed606" member_type: VOTER last_known_addr { host: "127.3.131.129" port: 41035 } }
I20260621 05:13:41.371097 19594 ts_tablet_manager.cc:1428] T 6a2f678342444e57bb2bccae6c6af0c9 P af09019aeb194a60b128a035658ed606: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260621 05:13:41.372218 19979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:13:41.367357 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.131.131:38877 (local address 127.0.0.1:50500)
0621 05:13:41.367591 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0621 05:13:41.367603 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0621 05:13:41.367657 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:13:41.367929 (+ 272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:13:41.367930 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:13:41.367932 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 05:13:41.367983 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:13:41.367985 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:13:41.369051 (+ 1066us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:13:41.369052 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:13:41.370334 (+ 1282us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:13:41.370335 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:13:41.371893 (+ 1558us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:13:41.371894 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:13:41.371966 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:13:41.371998 (+ 32us) client_negotiation.cc:769] Sending connection context
0621 05:13:41.372070 (+ 72us) client_negotiation.cc:240] Negotiation successful
0621 05:13:41.372122 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":212,"mutex_wait_us":8,"thread_start_us":120,"threads_started":1}
I20260621 05:13:41.372251 19981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:13:41.367437 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50500 (local address 127.3.131.131:38877)
0621 05:13:41.367692 (+ 255us) server_negotiation.cc:206] Beginning negotiation
0621 05:13:41.367697 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 05:13:41.367706 (+ 9us) server_negotiation.cc:407] Connection header received
0621 05:13:41.367745 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:13:41.367748 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:13:41.367763 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:13:41.367846 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 05:13:41.368080 (+ 234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:13:41.368966 (+ 886us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:13:41.370433 (+ 1467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:13:41.371785 (+ 1352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:13:41.371878 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:13:41.371960 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0621 05:13:41.372141 (+ 181us) server_negotiation.cc:299] Negotiation successful
0621 05:13:41.372183 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":220,"thread_start_us":103,"threads_started":1}
I20260621 05:13:41.372736 19709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:13:41.367242 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.131.130:44723 (local address 127.0.0.1:39552)
0621 05:13:41.367523 (+ 281us) negotiation.cc:107] Waiting for socket to connect
0621 05:13:41.367534 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0621 05:13:41.367577 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:13:41.367962 (+ 385us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:13:41.367963 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:13:41.367964 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 05:13:41.367992 (+ 28us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:13:41.367993 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:13:41.369228 (+ 1235us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:13:41.369229 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:13:41.370321 (+ 1092us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:13:41.370324 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:13:41.372447 (+ 2123us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:13:41.372448 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:13:41.372539 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:13:41.372545 (+ 6us) client_negotiation.cc:769] Sending connection context
0621 05:13:41.372607 (+ 62us) client_negotiation.cc:240] Negotiation successful
0621 05:13:41.372660 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":262,"mutex_wait_us":5}
I20260621 05:13:41.372771 19980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:13:41.367375 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39552 (local address 127.3.131.130:44723)
0621 05:13:41.367779 (+ 404us) server_negotiation.cc:206] Beginning negotiation
0621 05:13:41.367784 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 05:13:41.367793 (+ 9us) server_negotiation.cc:407] Connection header received
0621 05:13:41.367831 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:13:41.367833 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:13:41.367849 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:13:41.367963 (+ 114us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 05:13:41.368046 (+ 83us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:13:41.369136 (+ 1090us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:13:41.371357 (+ 2221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:13:41.372334 (+ 977us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:13:41.372439 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:13:41.372501 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0621 05:13:41.372634 (+ 133us) server_negotiation.cc:299] Negotiation successful
0621 05:13:41.372681 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":368,"thread_start_us":112,"threads_started":1}
I20260621 05:13:41.373219 19907 tablet_service.cc:1467] Processing CreateTablet for tablet 6a2f678342444e57bb2bccae6c6af0c9 (DEFAULT_TABLE table=test-table [id=5034638cf3c343a5a47465e07054b90f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 05:13:41.373265 19775 tablet_service.cc:1467] Processing CreateTablet for tablet 6a2f678342444e57bb2bccae6c6af0c9 (DEFAULT_TABLE table=test-table [id=5034638cf3c343a5a47465e07054b90f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 05:13:41.373624 19907 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6a2f678342444e57bb2bccae6c6af0c9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 05:13:41.373632 19775 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6a2f678342444e57bb2bccae6c6af0c9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 05:13:41.375454 19733 tablet_bootstrap.cc:492] T 6a2f678342444e57bb2bccae6c6af0c9 P 518697964a27412aa91818fcc615e6d5: Bootstrap starting.
I20260621 05:13:41.376338 19733 tablet_bootstrap.cc:654] T 6a2f678342444e57bb2bccae6c6af0c9 P 518697964a27412aa91818fcc615e6d5: Neither blocks nor log segments found. Creating new log.
I20260621 05:13:41.376673 19733 log.cc:824] T 6a2f678342444e57bb2bccae6c6af0c9 P 518697964a27412aa91818fcc615e6d5: Log is configured to *not* fsync() on all Append() calls
I20260621 05:13:41.377104 19983 tablet_bootstrap.cc:492] T 6a2f678342444e57bb2bccae6c6af0c9 P 245dfac09a1a4fcba9d4ed15c2387fc8: Bootstrap starting.
I20260621 05:13:41.377703 19983 tablet_bootstrap.cc:654] T 6a2f678342444e57bb2bccae6c6af0c9 P 245dfac09a1a4fcba9d4ed15c2387fc8: Neither blocks nor log segments found. Creating new log.
I20260621 05:13:41.377724 19733 tablet_bootstrap.cc:492] T 6a2f678342444e57bb2bccae6c6af0c9 P 518697964a27412aa91818fcc615e6d5: No bootstrap required, opened a new log
I20260621 05:13:41.377800 19733 ts_tablet_manager.cc:1397] T 6a2f678342444e57bb2bccae6c6af0c9 P 518697964a27412aa91818fcc615e6d5: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260621 05:13:41.378206 19983 log.cc:824] T 6a2f678342444e57bb2bccae6c6af0c9 P 245dfac09a1a4fcba9d4ed15c2387fc8: Log is configured to *not* fsync() on all Append() calls
I20260621 05:13:41.378894 19983 tablet_bootstrap.cc:492] T 6a2f678342444e57bb2bccae6c6af0c9 P 245dfac09a1a4fcba9d4ed15c2387fc8: No bootstrap required, opened a new log
I20260621 05:13:41.378947 19983 ts_tablet_manager.cc:1397] T 6a2f678342444e57bb2bccae6c6af0c9 P 245dfac09a1a4fcba9d4ed15c2387fc8: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260621 05:13:41.379420 19733 raft_consensus.cc:348] T 6a2f678342444e57bb2bccae6c6af0c9 P 518697964a27412aa91818fcc615e6d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "518697964a27412aa91818fcc615e6d5" member_type: VOTER last_known_addr { host: "127.3.131.130" port: 44723 } } peers { permanent_uuid: "245dfac09a1a4fcba9d4ed15c2387fc8" member_type: VOTER last_known_addr { host: "127.3.131.131" port: 38877 } } peers { permanent_uuid: "af09019aeb194a60b128a035658ed606" member_type: VOTER last_known_addr { host: "127.3.131.129" port: 41035 } }
I20260621 05:13:41.379578 19733 raft_consensus.cc:374] T 6a2f678342444e57bb2bccae6c6af0c9 P 518697964a27412aa91818fcc615e6d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 05:13:41.379623 19733 raft_consensus.cc:729] T 6a2f678342444e57bb2bccae6c6af0c9 P 518697964a27412aa91818fcc615e6d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 518697964a27412aa91818fcc615e6d5, State: Initialized, Role: FOLLOWER
I20260621 05:13:41.379776 19733 consensus_queue.cc:260] T 6a2f678342444e57bb2bccae6c6af0c9 P 518697964a27412aa91818fcc615e6d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "518697964a27412aa91818fcc615e6d5" member_type: VOTER last_known_addr { host: "127.3.131.130" port: 44723 } } peers { permanent_uuid: "245dfac09a1a4fcba9d4ed15c2387fc8" member_type: VOTER last_known_addr { host: "127.3.131.131" port: 38877 } } peers { permanent_uuid: "af09019aeb194a60b128a035658ed606" member_type: VOTER last_known_addr { host: "127.3.131.129" port: 41035 } }
I20260621 05:13:41.380096 19733 ts_tablet_manager.cc:1428] T 6a2f678342444e57bb2bccae6c6af0c9 P 518697964a27412aa91818fcc615e6d5: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260621 05:13:41.380118 19841 heartbeater.cc:503] Master 127.3.131.190:46697 was elected leader, sending a full tablet report...
I20260621 05:13:41.380715 19983 raft_consensus.cc:348] T 6a2f678342444e57bb2bccae6c6af0c9 P 245dfac09a1a4fcba9d4ed15c2387fc8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "518697964a27412aa91818fcc615e6d5" member_type: VOTER last_known_addr { host: "127.3.131.130" port: 44723 } } peers { permanent_uuid: "245dfac09a1a4fcba9d4ed15c2387fc8" member_type: VOTER last_known_addr { host: "127.3.131.131" port: 38877 } } peers { permanent_uuid: "af09019aeb194a60b128a035658ed606" member_type: VOTER last_known_addr { host: "127.3.131.129" port: 41035 } }
I20260621 05:13:41.380913 19983 raft_consensus.cc:374] T 6a2f678342444e57bb2bccae6c6af0c9 P 245dfac09a1a4fcba9d4ed15c2387fc8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 05:13:41.380950 19983 raft_consensus.cc:729] T 6a2f678342444e57bb2bccae6c6af0c9 P 245dfac09a1a4fcba9d4ed15c2387fc8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 245dfac09a1a4fcba9d4ed15c2387fc8, State: Initialized, Role: FOLLOWER
I20260621 05:13:41.381060 19983 consensus_queue.cc:260] T 6a2f678342444e57bb2bccae6c6af0c9 P 245dfac09a1a4fcba9d4ed15c2387fc8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "518697964a27412aa91818fcc615e6d5" member_type: VOTER last_known_addr { host: "127.3.131.130" port: 44723 } } peers { permanent_uuid: "245dfac09a1a4fcba9d4ed15c2387fc8" member_type: VOTER last_known_addr { host: "127.3.131.131" port: 38877 } } peers { permanent_uuid: "af09019aeb194a60b128a035658ed606" member_type: VOTER last_known_addr { host: "127.3.131.129" port: 41035 } }
I20260621 05:13:41.381361 19983 ts_tablet_manager.cc:1428] T 6a2f678342444e57bb2bccae6c6af0c9 P 245dfac09a1a4fcba9d4ed15c2387fc8: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260621 05:13:41.381392 19973 heartbeater.cc:503] Master 127.3.131.190:46697 was elected leader, sending a full tablet report...
W20260621 05:13:41.447293 19842 tablet.cc:2369] T 6a2f678342444e57bb2bccae6c6af0c9 P 518697964a27412aa91818fcc615e6d5: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260621 05:13:41.556380 19703 tablet.cc:2369] T 6a2f678342444e57bb2bccae6c6af0c9 P af09019aeb194a60b128a035658ed606: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260621 05:13:41.585377 19974 tablet.cc:2369] T 6a2f678342444e57bb2bccae6c6af0c9 P 245dfac09a1a4fcba9d4ed15c2387fc8: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260621 05:13:41.601236 19987 raft_consensus.cc:482] T 6a2f678342444e57bb2bccae6c6af0c9 P 245dfac09a1a4fcba9d4ed15c2387fc8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260621 05:13:41.601374 19987 raft_consensus.cc:504] T 6a2f678342444e57bb2bccae6c6af0c9 P 245dfac09a1a4fcba9d4ed15c2387fc8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "518697964a27412aa91818fcc615e6d5" member_type: VOTER last_known_addr { host: "127.3.131.130" port: 44723 } } peers { permanent_uuid: "245dfac09a1a4fcba9d4ed15c2387fc8" member_type: VOTER last_known_addr { host: "127.3.131.131" port: 38877 } } peers { permanent_uuid: "af09019aeb194a60b128a035658ed606" member_type: VOTER last_known_addr { host: "127.3.131.129" port: 41035 } }
I20260621 05:13:41.602068 19987 leader_election.cc:288] T 6a2f678342444e57bb2bccae6c6af0c9 P 245dfac09a1a4fcba9d4ed15c2387fc8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 518697964a27412aa91818fcc615e6d5 (127.3.131.130:44723), af09019aeb194a60b128a035658ed606 (127.3.131.129:41035)
I20260621 05:13:41.605927 19975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:13:41.602310 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.131.129:41035 (local address 127.3.131.131:53957)
0621 05:13:41.602367 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0621 05:13:41.602381 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0621 05:13:41.602433 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:13:41.602839 (+ 406us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:13:41.602840 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:13:41.602843 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 05:13:41.602921 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:13:41.602922 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:13:41.603780 (+ 858us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:13:41.603781 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:13:41.604631 (+ 850us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:13:41.604634 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:13:41.605624 (+ 990us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:13:41.605625 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:13:41.605713 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:13:41.605717 (+ 4us) client_negotiation.cc:769] Sending connection context
0621 05:13:41.605788 (+ 71us) client_negotiation.cc:240] Negotiation successful
0621 05:13:41.605836 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32}
I20260621 05:13:41.605947 19988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:13:41.602415 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.131.131:53957 (local address 127.3.131.129:41035)
0621 05:13:41.602618 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0621 05:13:41.602620 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:13:41.602629 (+ 9us) server_negotiation.cc:407] Connection header received
0621 05:13:41.602660 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:13:41.602660 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:13:41.602698 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:13:41.602776 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 05:13:41.603024 (+ 248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:13:41.603656 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:13:41.604737 (+ 1081us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:13:41.605541 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:13:41.605605 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:13:41.605676 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0621 05:13:41.605832 (+ 156us) server_negotiation.cc:299] Negotiation successful
0621 05:13:41.605883 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":182,"thread_start_us":94,"threads_started":1}
I20260621 05:13:41.606329 19652 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a2f678342444e57bb2bccae6c6af0c9" candidate_uuid: "245dfac09a1a4fcba9d4ed15c2387fc8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af09019aeb194a60b128a035658ed606" is_pre_election: true
I20260621 05:13:41.606458 19989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:13:41.602410 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.131.130:44723 (local address 127.3.131.131:42261)
0621 05:13:41.602590 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0621 05:13:41.602600 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0621 05:13:41.602657 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:13:41.603096 (+ 439us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:13:41.603097 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:13:41.603100 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 05:13:41.603198 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:13:41.603200 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:13:41.604058 (+ 858us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:13:41.604059 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:13:41.605151 (+ 1092us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:13:41.605154 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:13:41.606091 (+ 937us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:13:41.606092 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:13:41.606176 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:13:41.606180 (+ 4us) client_negotiation.cc:769] Sending connection context
0621 05:13:41.606242 (+ 62us) client_negotiation.cc:240] Negotiation successful
0621 05:13:41.606281 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":162,"thread_start_us":103,"threads_started":1}
I20260621 05:13:41.606534 19652 raft_consensus.cc:2454] T 6a2f678342444e57bb2bccae6c6af0c9 P af09019aeb194a60b128a035658ed606 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 245dfac09a1a4fcba9d4ed15c2387fc8 in term 0.
I20260621 05:13:41.606760 19980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:13:41.602494 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.131.131:42261 (local address 127.3.131.130:44723)
0621 05:13:41.602554 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0621 05:13:41.602557 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0621 05:13:41.602696 (+ 139us) server_negotiation.cc:407] Connection header received
0621 05:13:41.602728 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:13:41.602730 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:13:41.602740 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:13:41.602859 (+ 119us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 05:13:41.603335 (+ 476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:13:41.603936 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:13:41.605338 (+ 1402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:13:41.605972 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:13:41.606045 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:13:41.606055 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0621 05:13:41.606621 (+ 566us) server_negotiation.cc:299] Negotiation successful
0621 05:13:41.606672 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260621 05:13:41.606791 19863 leader_election.cc:302] T 6a2f678342444e57bb2bccae6c6af0c9 P 245dfac09a1a4fcba9d4ed15c2387fc8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 245dfac09a1a4fcba9d4ed15c2387fc8, af09019aeb194a60b128a035658ed606; no voters:
I20260621 05:13:41.606974 19987 raft_consensus.cc:2781] T 6a2f678342444e57bb2bccae6c6af0c9 P 245dfac09a1a4fcba9d4ed15c2387fc8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260621 05:13:41.607062 19987 raft_consensus.cc:482] T 6a2f678342444e57bb2bccae6c6af0c9 P 245dfac09a1a4fcba9d4ed15c2387fc8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260621 05:13:41.607105 19987 raft_consensus.cc:3037] T 6a2f678342444e57bb2bccae6c6af0c9 P 245dfac09a1a4fcba9d4ed15c2387fc8 [term 0 FOLLOWER]: Advancing to term 1
I20260621 05:13:41.607128 19795 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a2f678342444e57bb2bccae6c6af0c9" candidate_uuid: "245dfac09a1a4fcba9d4ed15c2387fc8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "518697964a27412aa91818fcc615e6d5" is_pre_election: true
I20260621 05:13:41.607326 19795 raft_consensus.cc:2454] T 6a2f678342444e57bb2bccae6c6af0c9 P 518697964a27412aa91818fcc615e6d5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 245dfac09a1a4fcba9d4ed15c2387fc8 in term 0.
I20260621 05:13:41.608045 19987 raft_consensus.cc:504] T 6a2f678342444e57bb2bccae6c6af0c9 P 245dfac09a1a4fcba9d4ed15c2387fc8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "518697964a27412aa91818fcc615e6d5" member_type: VOTER last_known_addr { host: "127.3.131.130" port: 44723 } } peers { permanent_uuid: "245dfac09a1a4fcba9d4ed15c2387fc8" member_type: VOTER last_known_addr { host: "127.3.131.131" port: 38877 } } peers { permanent_uuid: "af09019aeb194a60b128a035658ed606" member_type: VOTER last_known_addr { host: "127.3.131.129" port: 41035 } }
I20260621 05:13:41.608232 19987 leader_election.cc:288] T 6a2f678342444e57bb2bccae6c6af0c9 P 245dfac09a1a4fcba9d4ed15c2387fc8 [CANDIDATE]: Term 1 election: Requested vote from peers 518697964a27412aa91818fcc615e6d5 (127.3.131.130:44723), af09019aeb194a60b128a035658ed606 (127.3.131.129:41035)
I20260621 05:13:41.608424 19795 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a2f678342444e57bb2bccae6c6af0c9" candidate_uuid: "245dfac09a1a4fcba9d4ed15c2387fc8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "518697964a27412aa91818fcc615e6d5"
I20260621 05:13:41.608457 19652 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a2f678342444e57bb2bccae6c6af0c9" candidate_uuid: "245dfac09a1a4fcba9d4ed15c2387fc8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af09019aeb194a60b128a035658ed606"
I20260621 05:13:41.608521 19795 raft_consensus.cc:3037] T 6a2f678342444e57bb2bccae6c6af0c9 P 518697964a27412aa91818fcc615e6d5 [term 0 FOLLOWER]: Advancing to term 1
I20260621 05:13:41.608568 19652 raft_consensus.cc:3037] T 6a2f678342444e57bb2bccae6c6af0c9 P af09019aeb194a60b128a035658ed606 [term 0 FOLLOWER]: Advancing to term 1
I20260621 05:13:41.609335 19652 raft_consensus.cc:2454] T 6a2f678342444e57bb2bccae6c6af0c9 P af09019aeb194a60b128a035658ed606 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 245dfac09a1a4fcba9d4ed15c2387fc8 in term 1.
I20260621 05:13:41.609601 19795 raft_consensus.cc:2454] T 6a2f678342444e57bb2bccae6c6af0c9 P 518697964a27412aa91818fcc615e6d5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 245dfac09a1a4fcba9d4ed15c2387fc8 in term 1.
I20260621 05:13:41.609633 19863 leader_election.cc:302] T 6a2f678342444e57bb2bccae6c6af0c9 P 245dfac09a1a4fcba9d4ed15c2387fc8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 245dfac09a1a4fcba9d4ed15c2387fc8, af09019aeb194a60b128a035658ed606; no voters:
I20260621 05:13:41.610009 19987 raft_consensus.cc:2781] T 6a2f678342444e57bb2bccae6c6af0c9 P 245dfac09a1a4fcba9d4ed15c2387fc8 [term 1 FOLLOWER]: Leader election won for term 1
I20260621 05:13:41.610236 19987 raft_consensus.cc:686] T 6a2f678342444e57bb2bccae6c6af0c9 P 245dfac09a1a4fcba9d4ed15c2387fc8 [term 1 LEADER]: Becoming Leader. State: Replica: 245dfac09a1a4fcba9d4ed15c2387fc8, State: Running, Role: LEADER
I20260621 05:13:41.610420 19987 consensus_queue.cc:237] T 6a2f678342444e57bb2bccae6c6af0c9 P 245dfac09a1a4fcba9d4ed15c2387fc8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "518697964a27412aa91818fcc615e6d5" member_type: VOTER last_known_addr { host: "127.3.131.130" port: 44723 } } peers { permanent_uuid: "245dfac09a1a4fcba9d4ed15c2387fc8" member_type: VOTER last_known_addr { host: "127.3.131.131" port: 38877 } } peers { permanent_uuid: "af09019aeb194a60b128a035658ed606" member_type: VOTER last_known_addr { host: "127.3.131.129" port: 41035 } }
I20260621 05:13:41.611747 19514 catalog_manager.cc:5495] T 6a2f678342444e57bb2bccae6c6af0c9 P 245dfac09a1a4fcba9d4ed15c2387fc8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 245dfac09a1a4fcba9d4ed15c2387fc8 (127.3.131.131). New cstate: current_term: 1 leader_uuid: "245dfac09a1a4fcba9d4ed15c2387fc8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "518697964a27412aa91818fcc615e6d5" member_type: VOTER last_known_addr { host: "127.3.131.130" port: 44723 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "245dfac09a1a4fcba9d4ed15c2387fc8" member_type: VOTER last_known_addr { host: "127.3.131.131" port: 38877 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "af09019aeb194a60b128a035658ed606" member_type: VOTER last_known_addr { host: "127.3.131.129" port: 41035 } health_report { overall_health: UNKNOWN } } }
I20260621 05:13:41.658931 19981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:13:41.656507 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50510 (local address 127.3.131.131:38877)
0621 05:13:41.656576 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0621 05:13:41.656578 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:13:41.656587 (+ 9us) server_negotiation.cc:407] Connection header received
0621 05:13:41.656635 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:13:41.656636 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:13:41.656689 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:13:41.656773 (+ 84us) server_negotiation.cc:226] Negotiated authn=TOKEN
0621 05:13:41.656914 (+ 141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:13:41.657392 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:13:41.658138 (+ 746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:13:41.658340 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:13:41.658416 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:13:41.658575 (+ 159us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0621 05:13:41.658683 (+ 108us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0621 05:13:41.658769 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0621 05:13:41.658799 (+ 30us) server_negotiation.cc:299] Negotiation successful
0621 05:13:41.658820 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260621 05:13:41.660892 19652 raft_consensus.cc:1261] T 6a2f678342444e57bb2bccae6c6af0c9 P af09019aeb194a60b128a035658ed606 [term 1 FOLLOWER]: Refusing update from remote peer 245dfac09a1a4fcba9d4ed15c2387fc8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 05:13:41.660915 19795 raft_consensus.cc:1261] T 6a2f678342444e57bb2bccae6c6af0c9 P 518697964a27412aa91818fcc615e6d5 [term 1 FOLLOWER]: Refusing update from remote peer 245dfac09a1a4fcba9d4ed15c2387fc8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 05:13:41.661293 19987 consensus_queue.cc:1035] T 6a2f678342444e57bb2bccae6c6af0c9 P 245dfac09a1a4fcba9d4ed15c2387fc8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "af09019aeb194a60b128a035658ed606" member_type: VOTER last_known_addr { host: "127.3.131.129" port: 41035 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 05:13:41.661458 19990 consensus_queue.cc:1035] T 6a2f678342444e57bb2bccae6c6af0c9 P 245dfac09a1a4fcba9d4ed15c2387fc8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "518697964a27412aa91818fcc615e6d5" member_type: VOTER last_known_addr { host: "127.3.131.130" port: 44723 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 05:13:41.663584 19992 mvcc.cc:205] Tried to move back new op lower bound from 7299149093519724544 to 7299149093317033984. Current Snapshot: MvccSnapshot[applied={T|T < 7299149093519724544}]
I20260621 05:13:41.666517 19513 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41480:
table { table_name: "test-table" } modify_external_catalogs: true
I20260621 05:13:41.666647 19513 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41480:
table { table_name: "test-table" } modify_external_catalogs: true
I20260621 05:13:41.667472 19513 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 5b3269e323af422bb256795282d54866: Sending DeleteTablet for 3 replicas of tablet 6a2f678342444e57bb2bccae6c6af0c9
I20260621 05:13:41.667783 19622 tablet_service.cc:1514] Processing DeleteTablet for tablet 6a2f678342444e57bb2bccae6c6af0c9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 05:13:41 UTC) from {username='kudu', principal='kudu/127.3.131.190@KRBTEST.COM'} at 127.0.0.1:60660
I20260621 05:13:41.667871 19775 tablet_service.cc:1514] Processing DeleteTablet for tablet 6a2f678342444e57bb2bccae6c6af0c9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 05:13:41 UTC) from {username='kudu', principal='kudu/127.3.131.190@KRBTEST.COM'} at 127.0.0.1:39552
I20260621 05:13:41.668025 19907 tablet_service.cc:1514] Processing DeleteTablet for tablet 6a2f678342444e57bb2bccae6c6af0c9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 05:13:41 UTC) from {username='kudu', principal='kudu/127.3.131.190@KRBTEST.COM'} at 127.0.0.1:50500
I20260621 05:13:41.668188 20002 tablet_replica.cc:330] T 6a2f678342444e57bb2bccae6c6af0c9 P 518697964a27412aa91818fcc615e6d5: stopping tablet replica
I20260621 05:13:41.668254 3598 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWZOx2_/build/release/bin/kudu with pid 19552
I20260621 05:13:41.668262 20001 tablet_replica.cc:330] T 6a2f678342444e57bb2bccae6c6af0c9 P af09019aeb194a60b128a035658ed606: stopping tablet replica
I20260621 05:13:41.668326 20001 raft_consensus.cc:2229] T 6a2f678342444e57bb2bccae6c6af0c9 P af09019aeb194a60b128a035658ed606 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260621 05:13:41.668391 20001 raft_consensus.cc:2258] T 6a2f678342444e57bb2bccae6c6af0c9 P af09019aeb194a60b128a035658ed606 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 05:13:41.668488 20002 raft_consensus.cc:2229] T 6a2f678342444e57bb2bccae6c6af0c9 P 518697964a27412aa91818fcc615e6d5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260621 05:13:41.668646 20002 raft_consensus.cc:2258] T 6a2f678342444e57bb2bccae6c6af0c9 P 518697964a27412aa91818fcc615e6d5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 05:13:41.669152 20002 ts_tablet_manager.cc:1905] T 6a2f678342444e57bb2bccae6c6af0c9 P 518697964a27412aa91818fcc615e6d5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 05:13:41.671404 20002 ts_tablet_manager.cc:1918] T 6a2f678342444e57bb2bccae6c6af0c9 P 518697964a27412aa91818fcc615e6d5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260621 05:13:41.671478 20002 log.cc:1196] T 6a2f678342444e57bb2bccae6c6af0c9 P 518697964a27412aa91818fcc615e6d5: Deleting WAL directory at /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782018616648578-3598-0/minicluster-data/ts-1/wal/wals/6a2f678342444e57bb2bccae6c6af0c9
I20260621 05:13:41.671733 20002 ts_tablet_manager.cc:1939] T 6a2f678342444e57bb2bccae6c6af0c9 P 518697964a27412aa91818fcc615e6d5: Deleting consensus metadata
I20260621 05:13:41.673573 19498 catalog_manager.cc:4841] TS 518697964a27412aa91818fcc615e6d5 (127.3.131.130:44723): tablet 6a2f678342444e57bb2bccae6c6af0c9 (table test-table [id=5034638cf3c343a5a47465e07054b90f]) successfully deleted
W20260621 05:13:41.676086 19501 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.3.131.129:41035 (error 108)
I20260621 05:13:41.676532 20003 tablet_replica.cc:330] T 6a2f678342444e57bb2bccae6c6af0c9 P 245dfac09a1a4fcba9d4ed15c2387fc8: stopping tablet replica
I20260621 05:13:41.676680 20003 raft_consensus.cc:2229] T 6a2f678342444e57bb2bccae6c6af0c9 P 245dfac09a1a4fcba9d4ed15c2387fc8 [term 1 LEADER]: Raft consensus shutting down.
I20260621 05:13:41.676757 19709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:13:41.676608 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.131.129:41035 (local address 127.0.0.1:43508)
0621 05:13:41.676668 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0621 05:13:41.676720 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.131.129:41035: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
I20260621 05:13:41.676856 20003 raft_consensus.cc:2258] T 6a2f678342444e57bb2bccae6c6af0c9 P 245dfac09a1a4fcba9d4ed15c2387fc8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 05:13:41.677073 3598 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWZOx2_/build/release/bin/kudu with pid 19708
W20260621 05:13:41.676888 19501 catalog_manager.cc:4584] TS af09019aeb194a60b128a035658ed606 (127.3.131.129:41035): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 6a2f678342444e57bb2bccae6c6af0c9: Network error: Client connection negotiation failed: client connection to 127.3.131.129:41035: connect: Connection refused (error 111)
I20260621 05:13:41.677968 20003 ts_tablet_manager.cc:1905] T 6a2f678342444e57bb2bccae6c6af0c9 P 245dfac09a1a4fcba9d4ed15c2387fc8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 05:13:41.680543 20003 ts_tablet_manager.cc:1918] T 6a2f678342444e57bb2bccae6c6af0c9 P 245dfac09a1a4fcba9d4ed15c2387fc8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260621 05:13:41.680737 20003 log.cc:1196] T 6a2f678342444e57bb2bccae6c6af0c9 P 245dfac09a1a4fcba9d4ed15c2387fc8: Deleting WAL directory at /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782018616648578-3598-0/minicluster-data/ts-2/wal/wals/6a2f678342444e57bb2bccae6c6af0c9
I20260621 05:13:41.681142 20003 ts_tablet_manager.cc:1939] T 6a2f678342444e57bb2bccae6c6af0c9 P 245dfac09a1a4fcba9d4ed15c2387fc8: Deleting consensus metadata
I20260621 05:13:41.681797 19498 catalog_manager.cc:4841] TS 245dfac09a1a4fcba9d4ed15c2387fc8 (127.3.131.131:38877): tablet 6a2f678342444e57bb2bccae6c6af0c9 (table test-table [id=5034638cf3c343a5a47465e07054b90f]) successfully deleted
I20260621 05:13:41.683696 3598 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWZOx2_/build/release/bin/kudu with pid 19844
I20260621 05:13:41.688859 3598 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWZOx2_/build/release/bin/kudu with pid 19481
I20260621 05:14:01.694813 3598 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWZOx2_/build/release/bin/kudu
/tmp/dist-test-taskWZOx2_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782018616648578-3598-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782018616648578-3598-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782018616648578-3598-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782018616648578-3598-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.131.190:46697
--webserver_interface=127.3.131.190
--webserver_port=35139
--keytab_file=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782018616648578-3598-0/krb5kdc/kudu_127.3.131.190.keytab
--principal=kudu/127.3.131.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.131.148:44007
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.131.190:46697
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782018616648578-3598-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782018616648578-3598-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782018616648578-3598-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 05:14:01.791309 20005 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 05:14:01.791563 20005 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 05:14:01.791606 20005 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 05:14:01.792830 20005 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260621 05:14:01.792894 20005 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260621 05:14:01.792915 20005 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260621 05:14:01.792934 20005 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 05:14:01.792991 20005 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 05:14:01.793023 20005 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 05:14:01.793041 20005 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 05:14:01.793072 20005 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 05:14:01.794603 20005 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.131.148:44007
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782018616648578-3598-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782018616648578-3598-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.3.131.190:46697
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782018616648578-3598-0/krb5kdc/kudu_127.3.131.190.keytab
--principal=kudu/127.3.131.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.131.190:46697
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782018616648578-3598-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.131.190
--webserver_port=35139
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20005
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782018616648578-3598-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 21 Jun 2026 04:16:16 UTC on ccycloud.quasar-xycaha.root.comops.site
build id 4457881
I20260621 05:14:01.794900 20005 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 05:14:01.795140 20005 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260621 05:14:01.800745 20005 server_base.cc:1034] running on GCE node
W20260621 05:14:01.800715 20013 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:14:01.800833 20010 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:14:01.800715 20011 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:14:01.801239 20005 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 05:14:01.801503 20005 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 05:14:01.802655 20005 hybrid_clock.cc:648] HybridClock initialized: now 1782018841802640 us; error 31 us; skew 500 ppm
Jun 21 05:14:01 dist-test-slave-vrq1 krb5kdc[18933](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782018841, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.131.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 05:14:01.810273 20005 init.cc:377] Logged in from keytab as kudu/127.3.131.190@KRBTEST.COM (short username kudu)
I20260621 05:14:01.810853 20005 webserver.cc:460] Webserver started at http://127.3.131.190:35139/ using document root <none> and password file <none>
I20260621 05:14:01.811118 20005 fs_manager.cc:359] Metadata directory not provided
I20260621 05:14:01.811164 20005 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 05:14:01.812687 20005 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260621 05:14:01.813524 20020 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 05:14:01.813822 20005 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260621 05:14:01.813959 20005 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782018616648578-3598-0/minicluster-data/master-0/data,/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782018616648578-3598-0/minicluster-data/master-0/wal
uuid: "5b3269e323af422bb256795282d54866"
format_stamp: "Formatted at 2026-06-21 05:13:20 on dist-test-slave-vrq1"
server_key: "456833c8468b3a016c85b28e4daff374"
server_key_iv: "0b2b5d432553b57c179b0a4863527276"
server_key_version: "encryptionkey@0"
I20260621 05:14:01.814494 20005 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782018616648578-3598-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782018616648578-3598-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782018616648578-3598-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 05:14:01.823602 20005 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 05:14:01.826079 20005 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 05:14:01.826272 20005 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 05:14:01.830575 20005 rpc_server.cc:307] RPC server started. Bound to: 127.3.131.190:46697
I20260621 05:14:01.830612 20072 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.131.190:46697 every 8 connection(s)
I20260621 05:14:01.831051 20005 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782018616648578-3598-0/minicluster-data/master-0/data/info.pb
I20260621 05:14:01.832834 20073 sys_catalog.cc:263] Verifying existing consensus state
I20260621 05:14:01.833503 20073 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5b3269e323af422bb256795282d54866: Bootstrap starting.
I20260621 05:14:01.841301 3598 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWZOx2_/build/release/bin/kudu as pid 20005
I20260621 05:14:01.841719 3598 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWZOx2_/build/release/bin/kudu
/tmp/dist-test-taskWZOx2_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782018616648578-3598-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782018616648578-3598-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782018616648578-3598-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782018616648578-3598-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.131.129:41035
--local_ip_for_outbound_sockets=127.3.131.129
--tserver_master_addrs=127.3.131.190:46697
--webserver_port=34091
--webserver_interface=127.3.131.129
--keytab_file=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782018616648578-3598-0/krb5kdc/kudu_127.3.131.129.keytab
--principal=kudu/127.3.131.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.131.148:44007
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782018616648578-3598-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782018616648578-3598-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782018616648578-3598-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260621 05:14:01.843062 20073 log.cc:824] T 00000000000000000000000000000000 P 5b3269e323af422bb256795282d54866: Log is configured to *not* fsync() on all Append() calls
I20260621 05:14:01.845417 20073 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5b3269e323af422bb256795282d54866: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260621 05:14:01.845758 20073 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5b3269e323af422bb256795282d54866: Bootstrap complete.
I20260621 05:14:01.848198 20073 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5b3269e323af422bb256795282d54866 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b3269e323af422bb256795282d54866" member_type: VOTER last_known_addr { host: "127.3.131.190" port: 46697 } }
I20260621 05:14:01.848461 20073 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5b3269e323af422bb256795282d54866 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b3269e323af422bb256795282d54866, State: Initialized, Role: FOLLOWER
I20260621 05:14:01.848652 20073 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5b3269e323af422bb256795282d54866 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b3269e323af422bb256795282d54866" member_type: VOTER last_known_addr { host: "127.3.131.190" port: 46697 } }
I20260621 05:14:01.848737 20073 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5b3269e323af422bb256795282d54866 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260621 05:14:01.848811 20073 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5b3269e323af422bb256795282d54866 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260621 05:14:01.848861 20073 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5b3269e323af422bb256795282d54866 [term 2 FOLLOWER]: Advancing to term 3
I20260621 05:14:01.849908 20073 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5b3269e323af422bb256795282d54866 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b3269e323af422bb256795282d54866" member_type: VOTER last_known_addr { host: "127.3.131.190" port: 46697 } }
I20260621 05:14:01.850040 20073 leader_election.cc:302] T 00000000000000000000000000000000 P 5b3269e323af422bb256795282d54866 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5b3269e323af422bb256795282d54866; no voters:
I20260621 05:14:01.850250 20073 leader_election.cc:288] T 00000000000000000000000000000000 P 5b3269e323af422bb256795282d54866 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260621 05:14:01.850324 20077 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5b3269e323af422bb256795282d54866 [term 3 FOLLOWER]: Leader election won for term 3
I20260621 05:14:01.850534 20077 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5b3269e323af422bb256795282d54866 [term 3 LEADER]: Becoming Leader. State: Replica: 5b3269e323af422bb256795282d54866, State: Running, Role: LEADER
I20260621 05:14:01.850584 20073 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5b3269e323af422bb256795282d54866 [sys.catalog]: configured and running, proceeding with master startup.
I20260621 05:14:01.850656 20077 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5b3269e323af422bb256795282d54866 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b3269e323af422bb256795282d54866" member_type: VOTER last_known_addr { host: "127.3.131.190" port: 46697 } }
I20260621 05:14:01.850967 20077 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5b3269e323af422bb256795282d54866 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5b3269e323af422bb256795282d54866. Latest consensus state: current_term: 3 leader_uuid: "5b3269e323af422bb256795282d54866" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b3269e323af422bb256795282d54866" member_type: VOTER last_known_addr { host: "127.3.131.190" port: 46697 } } }
I20260621 05:14:01.851058 20077 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5b3269e323af422bb256795282d54866 [sys.catalog]: This master's current role is: LEADER
I20260621 05:14:01.850993 20078 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5b3269e323af422bb256795282d54866 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "5b3269e323af422bb256795282d54866" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b3269e323af422bb256795282d54866" member_type: VOTER last_known_addr { host: "127.3.131.190" port: 46697 } } }
I20260621 05:14:01.851289 20078 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5b3269e323af422bb256795282d54866 [sys.catalog]: This master's current role is: LEADER
I20260621 05:14:01.852685 20093 catalog_manager.cc:1259] Loaded cluster ID: dce3d0e84ae94f97814f61bb245e0d57
I20260621 05:14:01.852733 20093 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 5b3269e323af422bb256795282d54866: loading cluster ID for follower catalog manager: success
I20260621 05:14:01.853899 20093 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 5b3269e323af422bb256795282d54866: acquiring CA information for follower catalog manager: success
F20260621 05:14:01.854308 20093 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f2d138a79ad google::LogMessage::Fail() at ??:0
@ 0x7f2d138aa88a google::LogMessage::SendToLog() at ??:0
@ 0x7f2d138a74de google::LogMessage::Flush() at ??:0
@ 0x7f2d138a8a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f2d16014104 kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7f2d1b8006b9 kudu::master::CatalogManager::LoadTspkEntries() at ??:0
@ 0x7f2d1b800989 kudu::master::CatalogManager::PrepareFollowerTokenVerifier() at ??:0
@ 0x7f2d1b800e4e kudu::master::CatalogManager::PrepareFollower() at ??:0
@ 0x7f2d1b8038c2 kudu::master::CatalogManagerBgTasks::Run() at ??:0
@ 0x7f2d1576a954 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f2d165ce6db start_thread at ??:0
@ 0x7f2d123a561f clone at ??:0
W20260621 05:14:01.949738 20075 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 05:14:01.949992 20075 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 05:14:01.950023 20075 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 05:14:01.951665 20075 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 05:14:01.951732 20075 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 05:14:01.951766 20075 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.131.129
I20260621 05:14:01.953495 20075 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.131.148:44007
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782018616648578-3598-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782018616648578-3598-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782018616648578-3598-0/krb5kdc/kudu_127.3.131.129.keytab
--principal=kudu/127.3.131.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.131.129:41035
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782018616648578-3598-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.131.129
--webserver_port=34091
--tserver_master_addrs=127.3.131.190:46697
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20075
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.131.129
--log_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782018616648578-3598-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 21 Jun 2026 04:16:16 UTC on ccycloud.quasar-xycaha.root.comops.site
build id 4457881
I20260621 05:14:01.953962 20075 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 05:14:01.954272 20075 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 05:14:01.959539 20111 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:14:01.959717 20114 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:14:01.959539 20112 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:14:01.960179 20075 server_base.cc:1034] running on GCE node
I20260621 05:14:01.960427 20075 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 05:14:01.960748 20075 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 05:14:01.961942 20075 hybrid_clock.cc:648] HybridClock initialized: now 1782018841961929 us; error 28 us; skew 500 ppm
Jun 21 05:14:01 dist-test-slave-vrq1 krb5kdc[18933](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782018841, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.131.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 05:14:01.965147 20075 init.cc:377] Logged in from keytab as kudu/127.3.131.129@KRBTEST.COM (short username kudu)
I20260621 05:14:01.965757 20075 webserver.cc:460] Webserver started at http://127.3.131.129:34091/ using document root <none> and password file <none>
I20260621 05:14:01.966034 20075 fs_manager.cc:359] Metadata directory not provided
I20260621 05:14:01.966130 20075 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 05:14:01.967679 20075 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260621 05:14:01.968617 20121 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 05:14:01.968894 20075 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260621 05:14:01.968997 20075 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782018616648578-3598-0/minicluster-data/ts-0/data,/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782018616648578-3598-0/minicluster-data/ts-0/wal
uuid: "af09019aeb194a60b128a035658ed606"
format_stamp: "Formatted at 2026-06-21 05:13:20 on dist-test-slave-vrq1"
server_key: "3a7917f757ceb859a74758824cce2404"
server_key_iv: "6d67965e121b33fd4eed38e9a363377a"
server_key_version: "encryptionkey@0"
I20260621 05:14:01.969515 20075 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782018616648578-3598-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782018616648578-3598-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782018616648578-3598-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 05:14:01.980391 20075 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 05:14:01.983237 20075 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 05:14:01.983415 20075 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 05:14:01.983713 20075 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 05:14:01.984494 20128 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260621 05:14:01.986255 20075 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260621 05:14:01.986325 20075 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260621 05:14:01.986362 20075 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260621 05:14:01.987329 20075 ts_tablet_manager.cc:610] Registered 1 tablets
I20260621 05:14:01.987380 20075 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.001s sys 0.000s
I20260621 05:14:01.987500 20128 tablet_bootstrap.cc:492] T 6a2f678342444e57bb2bccae6c6af0c9 P af09019aeb194a60b128a035658ed606: Bootstrap starting.
I20260621 05:14:01.997156 20075 rpc_server.cc:307] RPC server started. Bound to: 127.3.131.129:41035
I20260621 05:14:01.997840 20235 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.131.129:41035 every 8 connection(s)
I20260621 05:14:01.998075 3598 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWZOx2_/build/release/bin/kudu as pid 20075
I20260621 05:14:01.998402 3598 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWZOx2_/build/release/bin/kudu
/tmp/dist-test-taskWZOx2_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782018616648578-3598-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782018616648578-3598-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782018616648578-3598-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782018616648578-3598-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.131.130:44723
--local_ip_for_outbound_sockets=127.3.131.130
--tserver_master_addrs=127.3.131.190:46697
--webserver_port=34509
--webserver_interface=127.3.131.130
--keytab_file=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782018616648578-3598-0/krb5kdc/kudu_127.3.131.130.keytab
--principal=kudu/127.3.131.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.131.148:44007
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782018616648578-3598-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782018616648578-3598-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782018616648578-3598-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260621 05:14:01.997812 20075 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782018616648578-3598-0/minicluster-data/ts-0/data/info.pb
I20260621 05:14:02.000682 20128 log.cc:824] T 6a2f678342444e57bb2bccae6c6af0c9 P af09019aeb194a60b128a035658ed606: Log is configured to *not* fsync() on all Append() calls
I20260621 05:14:02.003808 20128 tablet_bootstrap.cc:492] T 6a2f678342444e57bb2bccae6c6af0c9 P af09019aeb194a60b128a035658ed606: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260621 05:14:02.004181 20128 tablet_bootstrap.cc:492] T 6a2f678342444e57bb2bccae6c6af0c9 P af09019aeb194a60b128a035658ed606: Bootstrap complete.
I20260621 05:14:02.005493 20128 ts_tablet_manager.cc:1397] T 6a2f678342444e57bb2bccae6c6af0c9 P af09019aeb194a60b128a035658ed606: Time spent bootstrapping tablet: real 0.018s user 0.003s sys 0.014s
I20260621 05:14:02.007510 20128 raft_consensus.cc:348] T 6a2f678342444e57bb2bccae6c6af0c9 P af09019aeb194a60b128a035658ed606 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "518697964a27412aa91818fcc615e6d5" member_type: VOTER last_known_addr { host: "127.3.131.130" port: 44723 } } peers { permanent_uuid: "245dfac09a1a4fcba9d4ed15c2387fc8" member_type: VOTER last_known_addr { host: "127.3.131.131" port: 38877 } } peers { permanent_uuid: "af09019aeb194a60b128a035658ed606" member_type: VOTER last_known_addr { host: "127.3.131.129" port: 41035 } }
I20260621 05:14:02.007938 20128 raft_consensus.cc:729] T 6a2f678342444e57bb2bccae6c6af0c9 P af09019aeb194a60b128a035658ed606 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: af09019aeb194a60b128a035658ed606, State: Initialized, Role: FOLLOWER
I20260621 05:14:02.008114 20128 consensus_queue.cc:260] T 6a2f678342444e57bb2bccae6c6af0c9 P af09019aeb194a60b128a035658ed606 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "518697964a27412aa91818fcc615e6d5" member_type: VOTER last_known_addr { host: "127.3.131.130" port: 44723 } } peers { permanent_uuid: "245dfac09a1a4fcba9d4ed15c2387fc8" member_type: VOTER last_known_addr { host: "127.3.131.131" port: 38877 } } peers { permanent_uuid: "af09019aeb194a60b128a035658ed606" member_type: VOTER last_known_addr { host: "127.3.131.129" port: 41035 } }
I20260621 05:14:02.008462 20128 ts_tablet_manager.cc:1428] T 6a2f678342444e57bb2bccae6c6af0c9 P af09019aeb194a60b128a035658ed606: Time spent starting tablet: real 0.003s user 0.004s sys 0.000s
W20260621 05:14:02.078691 20238 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 05:14:01.999691 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.131.190:46697 (local address 127.3.131.129:59931)
0621 05:14:02.000111 (+ 420us) negotiation.cc:107] Waiting for socket to connect
0621 05:14:02.000173 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0621 05:14:02.001024 (+ 851us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:14:02.078476 (+ 77452us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.131.190:46697: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":307,"thread_start_us":120,"threads_started":1}
W20260621 05:14:02.079020 20236 heartbeater.cc:644] Failed to heartbeat to 127.3.131.190:46697 (0 consecutive failures): Network error: Failed to ping master at 127.3.131.190:46697: Client connection negotiation failed: client connection to 127.3.131.190:46697: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260621 05:14:02.079464 20238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:14:02.079344 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.131.190:46697 (local address 127.3.131.129:36455)
0621 05:14:02.079398 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0621 05:14:02.079443 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.131.190:46697: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
I20260621 05:14:02.079814 20238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:14:02.079720 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.131.190:46697 (local address 127.3.131.129:51235)
0621 05:14:02.079770 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0621 05:14:02.079794 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.131.190:46697: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260621 05:14:02.079926 20236 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260621 05:14:02.096652 20239 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 05:14:02.096906 20239 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 05:14:02.096948 20239 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 05:14:02.098133 20239 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 05:14:02.098201 20239 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 05:14:02.098249 20239 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.131.130
I20260621 05:14:02.100023 20239 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.131.148:44007
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782018616648578-3598-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782018616648578-3598-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782018616648578-3598-0/krb5kdc/kudu_127.3.131.130.keytab
--principal=kudu/127.3.131.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.131.130:44723
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782018616648578-3598-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.131.130
--webserver_port=34509
--tserver_master_addrs=127.3.131.190:46697
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20239
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.131.130
--log_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782018616648578-3598-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 21 Jun 2026 04:16:16 UTC on ccycloud.quasar-xycaha.root.comops.site
build id 4457881
I20260621 05:14:02.100442 20239 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 05:14:02.100838 20239 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 05:14:02.106691 20246 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:14:02.106719 20247 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:14:02.107311 20249 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:14:02.107373 20239 server_base.cc:1034] running on GCE node
I20260621 05:14:02.107589 20239 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 05:14:02.108045 20239 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 05:14:02.109273 20239 hybrid_clock.cc:648] HybridClock initialized: now 1782018842109258 us; error 66 us; skew 500 ppm
Jun 21 05:14:02 dist-test-slave-vrq1 krb5kdc[18933](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782018842, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.131.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 05:14:02.112210 20239 init.cc:377] Logged in from keytab as kudu/127.3.131.130@KRBTEST.COM (short username kudu)
I20260621 05:14:02.112725 20239 webserver.cc:460] Webserver started at http://127.3.131.130:34509/ using document root <none> and password file <none>
I20260621 05:14:02.112970 20239 fs_manager.cc:359] Metadata directory not provided
I20260621 05:14:02.113091 20239 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 05:14:02.114974 20239 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260621 05:14:02.115947 20256 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 05:14:02.116364 20239 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260621 05:14:02.116472 20239 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782018616648578-3598-0/minicluster-data/ts-1/data,/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782018616648578-3598-0/minicluster-data/ts-1/wal
uuid: "518697964a27412aa91818fcc615e6d5"
format_stamp: "Formatted at 2026-06-21 05:13:20 on dist-test-slave-vrq1"
server_key: "6a04f3ab56b30c3811a656e5d322d68f"
server_key_iv: "cf756f1d79126e3538eab75edec77821"
server_key_version: "encryptionkey@0"
I20260621 05:14:02.116917 20239 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782018616648578-3598-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782018616648578-3598-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782018616648578-3598-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 05:14:02.133699 20239 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 05:14:02.137140 20239 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 05:14:02.137283 20239 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 05:14:02.137557 20239 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 05:14:02.138024 20239 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 05:14:02.138109 20239 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 05:14:02.138211 20239 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 05:14:02.138260 20239 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 05:14:02.146019 20239 rpc_server.cc:307] RPC server started. Bound to: 127.3.131.130:44723
I20260621 05:14:02.146091 20369 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.131.130:44723 every 8 connection(s)
I20260621 05:14:02.146493 20239 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782018616648578-3598-0/minicluster-data/ts-1/data/info.pb
I20260621 05:14:02.146929 3598 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWZOx2_/build/release/bin/kudu as pid 20239
I20260621 05:14:02.147468 3598 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWZOx2_/build/release/bin/kudu
/tmp/dist-test-taskWZOx2_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782018616648578-3598-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782018616648578-3598-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782018616648578-3598-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782018616648578-3598-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.131.131:38877
--local_ip_for_outbound_sockets=127.3.131.131
--tserver_master_addrs=127.3.131.190:46697
--webserver_port=43527
--webserver_interface=127.3.131.131
--keytab_file=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782018616648578-3598-0/krb5kdc/kudu_127.3.131.131.keytab
--principal=kudu/127.3.131.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.131.148:44007
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782018616648578-3598-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782018616648578-3598-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782018616648578-3598-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260621 05:14:02.148175 20372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:14:02.147501 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.131.190:46697 (local address 127.3.131.130:60479)
0621 05:14:02.147700 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0621 05:14:02.147787 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.131.190:46697: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":103,"threads_started":1}
W20260621 05:14:02.148550 20370 heartbeater.cc:644] Failed to heartbeat to 127.3.131.190:46697 (0 consecutive failures): Network error: Failed to ping master at 127.3.131.190:46697: Client connection negotiation failed: client connection to 127.3.131.190:46697: connect: Connection refused (error 111)
I20260621 05:14:02.148938 20372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:14:02.148826 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.131.190:46697 (local address 127.3.131.130:57915)
0621 05:14:02.148880 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0621 05:14:02.148916 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.131.190:46697: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
I20260621 05:14:02.149323 20372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:14:02.149161 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.131.190:46697 (local address 127.3.131.130:57417)
0621 05:14:02.149225 (+ 64us) negotiation.cc:107] Waiting for socket to connect
0621 05:14:02.149287 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.131.190:46697: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260621 05:14:02.149533 20370 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260621 05:14:02.231462 20373 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 05:14:02.231729 20373 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 05:14:02.231773 20373 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 05:14:02.232970 20373 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 05:14:02.233057 20373 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 05:14:02.233101 20373 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.131.131
I20260621 05:14:02.234632 20373 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.131.148:44007
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782018616648578-3598-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782018616648578-3598-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782018616648578-3598-0/krb5kdc/kudu_127.3.131.131.keytab
--principal=kudu/127.3.131.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.131.131:38877
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782018616648578-3598-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.131.131
--webserver_port=43527
--tserver_master_addrs=127.3.131.190:46697
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20373
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.131.131
--log_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782018616648578-3598-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 21 Jun 2026 04:16:16 UTC on ccycloud.quasar-xycaha.root.comops.site
build id 4457881
I20260621 05:14:02.234920 20373 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 05:14:02.235203 20373 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 05:14:02.241423 20379 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:14:02.241413 20381 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:14:02.241628 20373 server_base.cc:1034] running on GCE node
W20260621 05:14:02.241413 20378 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:14:02.242141 20373 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 05:14:02.242437 20373 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 05:14:02.243656 20373 hybrid_clock.cc:648] HybridClock initialized: now 1782018842243618 us; error 46 us; skew 500 ppm
Jun 21 05:14:02 dist-test-slave-vrq1 krb5kdc[18933](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782018842, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.131.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 05:14:02.246438 20373 init.cc:377] Logged in from keytab as kudu/127.3.131.131@KRBTEST.COM (short username kudu)
I20260621 05:14:02.247041 20373 webserver.cc:460] Webserver started at http://127.3.131.131:43527/ using document root <none> and password file <none>
I20260621 05:14:02.247335 20373 fs_manager.cc:359] Metadata directory not provided
I20260621 05:14:02.247406 20373 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 05:14:02.248796 20373 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260621 05:14:02.249416 20388 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 05:14:02.249634 20373 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260621 05:14:02.249815 20373 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782018616648578-3598-0/minicluster-data/ts-2/data,/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782018616648578-3598-0/minicluster-data/ts-2/wal
uuid: "245dfac09a1a4fcba9d4ed15c2387fc8"
format_stamp: "Formatted at 2026-06-21 05:13:20 on dist-test-slave-vrq1"
server_key: "067ea0ddafaa5328ebc1fb466104a9e8"
server_key_iv: "417bfc56a463306fd6117f4085658fee"
server_key_version: "encryptionkey@0"
I20260621 05:14:02.250190 20373 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782018616648578-3598-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782018616648578-3598-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782018616648578-3598-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 05:14:02.277576 20373 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 05:14:02.280021 20373 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 05:14:02.280213 20373 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 05:14:02.280478 20373 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 05:14:02.280884 20373 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 05:14:02.280944 20373 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 05:14:02.280980 20373 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 05:14:02.281023 20373 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 05:14:02.287201 20373 rpc_server.cc:307] RPC server started. Bound to: 127.3.131.131:38877
I20260621 05:14:02.287266 20501 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.131.131:38877 every 8 connection(s)
I20260621 05:14:02.287655 20373 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782018616648578-3598-0/minicluster-data/ts-2/data/info.pb
I20260621 05:14:02.288830 20504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:14:02.288405 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.131.190:46697 (local address 127.3.131.131:54693)
0621 05:14:02.288575 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0621 05:14:02.288659 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.131.190:46697: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":71,"threads_started":1}
W20260621 05:14:02.289371 20502 heartbeater.cc:644] Failed to heartbeat to 127.3.131.190:46697 (0 consecutive failures): Network error: Failed to ping master at 127.3.131.190:46697: Client connection negotiation failed: client connection to 127.3.131.190:46697: connect: Connection refused (error 111)
I20260621 05:14:02.289712 20504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:14:02.289611 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.131.190:46697 (local address 127.3.131.131:42435)
0621 05:14:02.289660 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0621 05:14:02.289692 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.131.190:46697: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
I20260621 05:14:02.290103 20504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:14:02.289977 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.131.190:46697 (local address 127.3.131.131:41917)
0621 05:14:02.290044 (+ 67us) negotiation.cc:107] Waiting for socket to connect
0621 05:14:02.290074 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.131.190:46697: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260621 05:14:02.290292 20502 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260621 05:14:02.294634 3598 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWZOx2_/build/release/bin/kudu as pid 20373
I20260621 05:14:03.080667 20506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:14:03.080385 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.131.190:46697 (local address 127.3.131.129:43083)
0621 05:14:03.080538 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0621 05:14:03.080597 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.131.190:46697: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":71,"threads_started":1}
I20260621 05:14:03.150169 20507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:14:03.149910 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.131.190:46697 (local address 127.3.131.130:59131)
0621 05:14:03.150057 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0621 05:14:03.150111 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.131.190:46697: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":61,"threads_started":1}
I20260621 05:14:03.252058 20508 raft_consensus.cc:482] T 6a2f678342444e57bb2bccae6c6af0c9 P af09019aeb194a60b128a035658ed606 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260621 05:14:03.252226 20508 raft_consensus.cc:504] T 6a2f678342444e57bb2bccae6c6af0c9 P af09019aeb194a60b128a035658ed606 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "518697964a27412aa91818fcc615e6d5" member_type: VOTER last_known_addr { host: "127.3.131.130" port: 44723 } } peers { permanent_uuid: "245dfac09a1a4fcba9d4ed15c2387fc8" member_type: VOTER last_known_addr { host: "127.3.131.131" port: 38877 } } peers { permanent_uuid: "af09019aeb194a60b128a035658ed606" member_type: VOTER last_known_addr { host: "127.3.131.129" port: 41035 } }
I20260621 05:14:03.252596 20508 leader_election.cc:288] T 6a2f678342444e57bb2bccae6c6af0c9 P af09019aeb194a60b128a035658ed606 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 518697964a27412aa91818fcc615e6d5 (127.3.131.130:44723), 245dfac09a1a4fcba9d4ed15c2387fc8 (127.3.131.131:38877)
Jun 21 05:14:03 dist-test-slave-vrq1 krb5kdc[18933](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782018841, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.131.129@KRBTEST.COM for kudu/127.3.131.130@KRBTEST.COM
Jun 21 05:14:03 dist-test-slave-vrq1 krb5kdc[18933](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782018841, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.131.129@KRBTEST.COM for kudu/127.3.131.131@KRBTEST.COM
I20260621 05:14:03.264477 20506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:14:03.252840 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.131.130:44723 (local address 127.3.131.129:39455)
0621 05:14:03.252918 (+ 78us) negotiation.cc:107] Waiting for socket to connect
0621 05:14:03.252933 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0621 05:14:03.253011 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:14:03.254146 (+ 1135us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:14:03.254216 (+ 70us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:14:03.254705 (+ 489us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:14:03.254916 (+ 211us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:14:03.254923 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:14:03.256141 (+ 1218us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:14:03.256142 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:14:03.256973 (+ 831us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:14:03.256976 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:14:03.257589 (+ 613us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:14:03.257590 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:14:03.257707 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:14:03.258396 (+ 689us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 05:14:03.258405 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:14:03.260163 (+ 1758us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 05:14:03.262460 (+ 2297us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:14:03.262470 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:14:03.262473 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:14:03.262800 (+ 327us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:14:03.263154 (+ 354us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:14:03.263154 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:14:03.263155 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:14:03.263349 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:14:03.263881 (+ 532us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 05:14:03.263888 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 05:14:03.264044 (+ 156us) client_negotiation.cc:769] Sending connection context
0621 05:14:03.264158 (+ 114us) client_negotiation.cc:240] Negotiation successful
0621 05:14:03.264362 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":50,"mutex_wait_us":48}
I20260621 05:14:03.264695 20510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:14:03.253022 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.131.129:39455 (local address 127.3.131.130:44723)
0621 05:14:03.253206 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0621 05:14:03.253219 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0621 05:14:03.253254 (+ 35us) server_negotiation.cc:407] Connection header received
0621 05:14:03.253910 (+ 656us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:14:03.253913 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:14:03.253952 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:14:03.254103 (+ 151us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:14:03.255054 (+ 951us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:14:03.256020 (+ 966us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:14:03.257083 (+ 1063us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:14:03.257491 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:14:03.257616 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:14:03.260322 (+ 2706us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:14:03.260327 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:14:03.260327 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:14:03.260342 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:14:03.262359 (+ 2017us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:14:03.262927 (+ 568us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:14:03.262931 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:14:03.262932 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:14:03.263065 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:14:03.263424 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:14:03.263425 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:14:03.263425 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:14:03.263793 (+ 368us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:14:03.263859 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0621 05:14:03.264222 (+ 363us) server_negotiation.cc:299] Negotiation successful
0621 05:14:03.264501 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":144,"thread_start_us":120,"threads_started":1}
I20260621 05:14:03.264704 20511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:14:03.253080 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.131.129:36935 (local address 127.3.131.131:38877)
0621 05:14:03.253267 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0621 05:14:03.253276 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0621 05:14:03.253295 (+ 19us) server_negotiation.cc:407] Connection header received
0621 05:14:03.253854 (+ 559us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:14:03.253858 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:14:03.253940 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:14:03.254103 (+ 163us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:14:03.255045 (+ 942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:14:03.256015 (+ 970us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:14:03.257076 (+ 1061us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:14:03.257490 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:14:03.257616 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:14:03.260767 (+ 3151us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:14:03.260772 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:14:03.260773 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:14:03.260786 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:14:03.262846 (+ 2060us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:14:03.263296 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:14:03.263300 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:14:03.263300 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:14:03.263422 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:14:03.263697 (+ 275us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:14:03.263699 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:14:03.263699 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:14:03.264035 (+ 336us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:14:03.264148 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0621 05:14:03.264445 (+ 297us) server_negotiation.cc:299] Negotiation successful
0621 05:14:03.264592 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":146,"thread_start_us":82,"threads_started":1}
I20260621 05:14:03.264612 20509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:14:03.252996 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.131.131:38877 (local address 127.3.131.129:36935)
0621 05:14:03.253142 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0621 05:14:03.253151 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0621 05:14:03.253204 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:14:03.254254 (+ 1050us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:14:03.254255 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:14:03.254705 (+ 450us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:14:03.254916 (+ 211us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:14:03.254922 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:14:03.256135 (+ 1213us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:14:03.256136 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:14:03.256973 (+ 837us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:14:03.256976 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:14:03.257592 (+ 616us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:14:03.257593 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:14:03.257708 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:14:03.258412 (+ 704us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 05:14:03.258416 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:14:03.260632 (+ 2216us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 05:14:03.262938 (+ 2306us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:14:03.262940 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:14:03.262940 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:14:03.263170 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:14:03.263491 (+ 321us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:14:03.263492 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:14:03.263492 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:14:03.263553 (+ 61us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:14:03.264121 (+ 568us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 05:14:03.264122 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 05:14:03.264187 (+ 65us) client_negotiation.cc:769] Sending connection context
0621 05:14:03.264391 (+ 204us) client_negotiation.cc:240] Negotiation successful
0621 05:14:03.264513 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":131,"mutex_wait_us":37,"thread_start_us":56,"threads_started":1}
W20260621 05:14:03.266155 20122 leader_election.cc:341] T 6a2f678342444e57bb2bccae6c6af0c9 P af09019aeb194a60b128a035658ed606 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 518697964a27412aa91818fcc615e6d5 (127.3.131.130:44723): Not found: Tablet not found: 6a2f678342444e57bb2bccae6c6af0c9
W20260621 05:14:03.266301 20122 leader_election.cc:341] T 6a2f678342444e57bb2bccae6c6af0c9 P af09019aeb194a60b128a035658ed606 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 245dfac09a1a4fcba9d4ed15c2387fc8 (127.3.131.131:38877): Not found: Tablet not found: 6a2f678342444e57bb2bccae6c6af0c9
I20260621 05:14:03.266360 20122 leader_election.cc:302] T 6a2f678342444e57bb2bccae6c6af0c9 P af09019aeb194a60b128a035658ed606 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af09019aeb194a60b128a035658ed606; no voters: 245dfac09a1a4fcba9d4ed15c2387fc8, 518697964a27412aa91818fcc615e6d5
I20260621 05:14:03.266599 20508 raft_consensus.cc:2726] T 6a2f678342444e57bb2bccae6c6af0c9 P af09019aeb194a60b128a035658ed606 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260621 05:14:03.291002 20512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:14:03.290734 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.131.190:46697 (local address 127.3.131.131:40277)
0621 05:14:03.290899 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0621 05:14:03.290961 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.131.190:46697: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":82,"threads_started":1}
I20260621 05:14:04.081610 20513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:14:04.081373 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.131.190:46697 (local address 127.3.131.129:59003)
0621 05:14:04.081514 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0621 05:14:04.081561 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.131.190:46697: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":68,"threads_started":1}
I20260621 05:14:04.151054 20514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:14:04.150798 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.131.190:46697 (local address 127.3.131.130:40675)
0621 05:14:04.150957 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0621 05:14:04.151007 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.131.190:46697: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":65,"threads_started":1}
I20260621 05:14:04.291811 20515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:14:04.291544 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.131.190:46697 (local address 127.3.131.131:50175)
0621 05:14:04.291720 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0621 05:14:04.291765 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.131.190:46697: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":151,"thread_start_us":64,"threads_started":1}
I20260621 05:14:05.082572 20516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:14:05.082308 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.131.190:46697 (local address 127.3.131.129:38237)
0621 05:14:05.082476 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0621 05:14:05.082519 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.131.190:46697: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":70,"threads_started":1}
I20260621 05:14:05.151966 20517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:14:05.151714 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.131.190:46697 (local address 127.3.131.130:41715)
0621 05:14:05.151873 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0621 05:14:05.151919 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.131.190:46697: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":73,"threads_started":1}
I20260621 05:14:05.239977 20518 raft_consensus.cc:482] T 6a2f678342444e57bb2bccae6c6af0c9 P af09019aeb194a60b128a035658ed606 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260621 05:14:05.240088 20518 raft_consensus.cc:504] T 6a2f678342444e57bb2bccae6c6af0c9 P af09019aeb194a60b128a035658ed606 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "518697964a27412aa91818fcc615e6d5" member_type: VOTER last_known_addr { host: "127.3.131.130" port: 44723 } } peers { permanent_uuid: "245dfac09a1a4fcba9d4ed15c2387fc8" member_type: VOTER last_known_addr { host: "127.3.131.131" port: 38877 } } peers { permanent_uuid: "af09019aeb194a60b128a035658ed606" member_type: VOTER last_known_addr { host: "127.3.131.129" port: 41035 } }
I20260621 05:14:05.240272 20518 leader_election.cc:288] T 6a2f678342444e57bb2bccae6c6af0c9 P af09019aeb194a60b128a035658ed606 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 518697964a27412aa91818fcc615e6d5 (127.3.131.130:44723), 245dfac09a1a4fcba9d4ed15c2387fc8 (127.3.131.131:38877)
W20260621 05:14:05.240962 20122 leader_election.cc:341] T 6a2f678342444e57bb2bccae6c6af0c9 P af09019aeb194a60b128a035658ed606 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 245dfac09a1a4fcba9d4ed15c2387fc8 (127.3.131.131:38877): Not found: Tablet not found: 6a2f678342444e57bb2bccae6c6af0c9
W20260621 05:14:05.241037 20122 leader_election.cc:341] T 6a2f678342444e57bb2bccae6c6af0c9 P af09019aeb194a60b128a035658ed606 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 518697964a27412aa91818fcc615e6d5 (127.3.131.130:44723): Not found: Tablet not found: 6a2f678342444e57bb2bccae6c6af0c9
I20260621 05:14:05.241055 20122 leader_election.cc:302] T 6a2f678342444e57bb2bccae6c6af0c9 P af09019aeb194a60b128a035658ed606 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af09019aeb194a60b128a035658ed606; no voters: 245dfac09a1a4fcba9d4ed15c2387fc8, 518697964a27412aa91818fcc615e6d5
I20260621 05:14:05.241142 20518 raft_consensus.cc:2726] T 6a2f678342444e57bb2bccae6c6af0c9 P af09019aeb194a60b128a035658ed606 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260621 05:14:05.292806 20519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:14:05.292547 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.131.190:46697 (local address 127.3.131.131:53131)
0621 05:14:05.292707 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0621 05:14:05.292760 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.131.190:46697: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":69,"threads_started":1}
I20260621 05:14:06.083503 20520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:14:06.083246 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.131.190:46697 (local address 127.3.131.129:40327)
0621 05:14:06.083407 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0621 05:14:06.083456 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.131.190:46697: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":74,"threads_started":1}
I20260621 05:14:06.152962 20521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:14:06.152686 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.131.190:46697 (local address 127.3.131.130:43363)
0621 05:14:06.152866 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0621 05:14:06.152913 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.131.190:46697: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":158,"thread_start_us":74,"threads_started":1}
I20260621 05:14:06.293633 20522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:14:06.293404 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.131.190:46697 (local address 127.3.131.131:48663)
0621 05:14:06.293546 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0621 05:14:06.293587 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.131.190:46697: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":63,"threads_started":1}
I20260621 05:14:07.084507 20523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:14:07.084239 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.131.190:46697 (local address 127.3.131.129:50753)
0621 05:14:07.084405 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0621 05:14:07.084454 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.131.190:46697: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":72,"threads_started":1}
I20260621 05:14:07.153925 20524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:14:07.153689 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.131.190:46697 (local address 127.3.131.130:34101)
0621 05:14:07.153836 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0621 05:14:07.153879 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.131.190:46697: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":62,"threads_started":1}
I20260621 05:14:07.294483 20525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:14:07.294267 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.131.190:46697 (local address 127.3.131.131:46497)
0621 05:14:07.294396 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0621 05:14:07.294435 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.131.190:46697: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":87,"threads_started":1}
I20260621 05:14:07.914578 20526 raft_consensus.cc:482] T 6a2f678342444e57bb2bccae6c6af0c9 P af09019aeb194a60b128a035658ed606 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260621 05:14:07.914690 20526 raft_consensus.cc:504] T 6a2f678342444e57bb2bccae6c6af0c9 P af09019aeb194a60b128a035658ed606 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "518697964a27412aa91818fcc615e6d5" member_type: VOTER last_known_addr { host: "127.3.131.130" port: 44723 } } peers { permanent_uuid: "245dfac09a1a4fcba9d4ed15c2387fc8" member_type: VOTER last_known_addr { host: "127.3.131.131" port: 38877 } } peers { permanent_uuid: "af09019aeb194a60b128a035658ed606" member_type: VOTER last_known_addr { host: "127.3.131.129" port: 41035 } }
I20260621 05:14:07.914921 20526 leader_election.cc:288] T 6a2f678342444e57bb2bccae6c6af0c9 P af09019aeb194a60b128a035658ed606 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 518697964a27412aa91818fcc615e6d5 (127.3.131.130:44723), 245dfac09a1a4fcba9d4ed15c2387fc8 (127.3.131.131:38877)
W20260621 05:14:07.915436 20122 leader_election.cc:341] T 6a2f678342444e57bb2bccae6c6af0c9 P af09019aeb194a60b128a035658ed606 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 518697964a27412aa91818fcc615e6d5 (127.3.131.130:44723): Not found: Tablet not found: 6a2f678342444e57bb2bccae6c6af0c9
W20260621 05:14:07.915532 20122 leader_election.cc:341] T 6a2f678342444e57bb2bccae6c6af0c9 P af09019aeb194a60b128a035658ed606 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 245dfac09a1a4fcba9d4ed15c2387fc8 (127.3.131.131:38877): Not found: Tablet not found: 6a2f678342444e57bb2bccae6c6af0c9
I20260621 05:14:07.915606 20122 leader_election.cc:302] T 6a2f678342444e57bb2bccae6c6af0c9 P af09019aeb194a60b128a035658ed606 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af09019aeb194a60b128a035658ed606; no voters: 245dfac09a1a4fcba9d4ed15c2387fc8, 518697964a27412aa91818fcc615e6d5
I20260621 05:14:07.915733 20526 raft_consensus.cc:2726] T 6a2f678342444e57bb2bccae6c6af0c9 P af09019aeb194a60b128a035658ed606 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260621 05:14:08.085690 20527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:14:08.085366 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.131.190:46697 (local address 127.3.131.129:43487)
0621 05:14:08.085582 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0621 05:14:08.085636 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.131.190:46697: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":193,"thread_start_us":93,"threads_started":1}
I20260621 05:14:08.154809 20528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:14:08.154560 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.131.190:46697 (local address 127.3.131.130:49639)
0621 05:14:08.154716 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0621 05:14:08.154764 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.131.190:46697: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":84,"threads_started":1}
I20260621 05:14:08.295455 20529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:14:08.295120 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.131.190:46697 (local address 127.3.131.131:51165)
0621 05:14:08.295356 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0621 05:14:08.295405 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.131.190:46697: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":206,"thread_start_us":164,"threads_started":1}
I20260621 05:14:09.086628 20530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:14:09.086319 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.131.190:46697 (local address 127.3.131.129:42223)
0621 05:14:09.086528 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0621 05:14:09.086580 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.131.190:46697: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":108,"threads_started":1}
I20260621 05:14:09.155754 20531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:14:09.155495 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.131.190:46697 (local address 127.3.131.130:35239)
0621 05:14:09.155657 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0621 05:14:09.155692 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.131.190:46697: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":83,"threads_started":1}
I20260621 05:14:09.296428 20532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:14:09.296115 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.131.190:46697 (local address 127.3.131.131:35685)
0621 05:14:09.296297 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0621 05:14:09.296353 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.131.190:46697: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":152,"thread_start_us":73,"threads_started":1}
I20260621 05:14:09.756356 20533 raft_consensus.cc:482] T 6a2f678342444e57bb2bccae6c6af0c9 P af09019aeb194a60b128a035658ed606 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260621 05:14:09.756460 20533 raft_consensus.cc:504] T 6a2f678342444e57bb2bccae6c6af0c9 P af09019aeb194a60b128a035658ed606 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "518697964a27412aa91818fcc615e6d5" member_type: VOTER last_known_addr { host: "127.3.131.130" port: 44723 } } peers { permanent_uuid: "245dfac09a1a4fcba9d4ed15c2387fc8" member_type: VOTER last_known_addr { host: "127.3.131.131" port: 38877 } } peers { permanent_uuid: "af09019aeb194a60b128a035658ed606" member_type: VOTER last_known_addr { host: "127.3.131.129" port: 41035 } }
I20260621 05:14:09.756711 20533 leader_election.cc:288] T 6a2f678342444e57bb2bccae6c6af0c9 P af09019aeb194a60b128a035658ed606 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 518697964a27412aa91818fcc615e6d5 (127.3.131.130:44723), 245dfac09a1a4fcba9d4ed15c2387fc8 (127.3.131.131:38877)
W20260621 05:14:09.757157 20122 leader_election.cc:341] T 6a2f678342444e57bb2bccae6c6af0c9 P af09019aeb194a60b128a035658ed606 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 518697964a27412aa91818fcc615e6d5 (127.3.131.130:44723): Not found: Tablet not found: 6a2f678342444e57bb2bccae6c6af0c9
W20260621 05:14:09.757258 20122 leader_election.cc:341] T 6a2f678342444e57bb2bccae6c6af0c9 P af09019aeb194a60b128a035658ed606 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 245dfac09a1a4fcba9d4ed15c2387fc8 (127.3.131.131:38877): Not found: Tablet not found: 6a2f678342444e57bb2bccae6c6af0c9
I20260621 05:14:09.757296 20122 leader_election.cc:302] T 6a2f678342444e57bb2bccae6c6af0c9 P af09019aeb194a60b128a035658ed606 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af09019aeb194a60b128a035658ed606; no voters: 245dfac09a1a4fcba9d4ed15c2387fc8, 518697964a27412aa91818fcc615e6d5
I20260621 05:14:09.757409 20533 raft_consensus.cc:2726] T 6a2f678342444e57bb2bccae6c6af0c9 P af09019aeb194a60b128a035658ed606 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260621 05:14:10.087491 20534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:14:10.087232 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.131.190:46697 (local address 127.3.131.129:45129)
0621 05:14:10.087387 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0621 05:14:10.087433 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.131.190:46697: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":71,"threads_started":1}
I20260621 05:14:10.156540 20535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:14:10.156308 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.131.190:46697 (local address 127.3.131.130:37541)
0621 05:14:10.156446 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0621 05:14:10.156493 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.131.190:46697: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":60,"threads_started":1}
I20260621 05:14:10.297339 20536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:14:10.297053 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.131.190:46697 (local address 127.3.131.131:47393)
0621 05:14:10.297242 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0621 05:14:10.297291 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.131.190:46697: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":166,"thread_start_us":73,"threads_started":1}
I20260621 05:14:11.088320 20537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:14:11.088077 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.131.190:46697 (local address 127.3.131.129:44967)
0621 05:14:11.088215 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0621 05:14:11.088265 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.131.190:46697: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":71,"threads_started":1}
I20260621 05:14:11.157338 20538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:14:11.157083 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.131.190:46697 (local address 127.3.131.130:57973)
0621 05:14:11.157242 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0621 05:14:11.157285 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.131.190:46697: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":63,"threads_started":1}
I20260621 05:14:11.298246 20539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:14:11.297926 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.131.190:46697 (local address 127.3.131.131:46833)
0621 05:14:11.298117 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0621 05:14:11.298181 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.131.190:46697: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":72,"threads_started":1}
I20260621 05:14:12.089166 20540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:14:12.088916 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.131.190:46697 (local address 127.3.131.129:45627)
0621 05:14:12.089076 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0621 05:14:12.089119 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.131.190:46697: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":79,"threads_started":1}
I20260621 05:14:12.158250 20541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:14:12.157983 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.131.190:46697 (local address 127.3.131.130:38805)
0621 05:14:12.158147 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0621 05:14:12.158193 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.131.190:46697: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":89,"threads_started":1}
I20260621 05:14:12.299077 20542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:14:12.298831 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.131.190:46697 (local address 127.3.131.131:57097)
0621 05:14:12.298995 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0621 05:14:12.299031 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.131.190:46697: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":76,"threads_started":1}
I20260621 05:14:13.090021 20543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:14:13.089754 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.131.190:46697 (local address 127.3.131.129:44629)
0621 05:14:13.089928 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0621 05:14:13.089974 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.131.190:46697: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":151,"thread_start_us":85,"threads_started":1}
I20260621 05:14:13.159155 20544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:14:13.158866 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.131.190:46697 (local address 127.3.131.130:45077)
0621 05:14:13.159044 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0621 05:14:13.159097 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.131.190:46697: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":156,"thread_start_us":75,"threads_started":1}
I20260621 05:14:13.300019 20545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:14:13.299724 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.131.190:46697 (local address 127.3.131.131:34703)
0621 05:14:13.299902 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0621 05:14:13.299957 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.131.190:46697: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":91,"threads_started":1}
I20260621 05:14:14.090997 20546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:14:14.090711 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.131.190:46697 (local address 127.3.131.129:54081)
0621 05:14:14.090892 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0621 05:14:14.090948 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.131.190:46697: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":154,"thread_start_us":86,"threads_started":1}
I20260621 05:14:14.160140 20547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:14:14.159838 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.131.190:46697 (local address 127.3.131.130:40219)
0621 05:14:14.160001 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0621 05:14:14.160091 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.131.190:46697: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":80,"threads_started":1}
I20260621 05:14:14.300945 20548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:14:14.300578 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.131.190:46697 (local address 127.3.131.131:34183)
0621 05:14:14.300841 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0621 05:14:14.300888 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.131.190:46697: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":206,"thread_start_us":78,"threads_started":1}
I20260621 05:14:15.091996 20549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:14:15.091736 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.131.190:46697 (local address 127.3.131.129:59679)
0621 05:14:15.091901 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0621 05:14:15.091949 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.131.190:46697: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":79,"threads_started":1}
I20260621 05:14:15.161146 20550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:14:15.160870 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.131.190:46697 (local address 127.3.131.130:59473)
0621 05:14:15.161024 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0621 05:14:15.161072 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.131.190:46697: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":65,"threads_started":1}
I20260621 05:14:15.301771 20551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:14:15.301521 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.131.190:46697 (local address 127.3.131.131:52909)
0621 05:14:15.301660 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0621 05:14:15.301711 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.131.190:46697: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":77,"threads_started":1}
I20260621 05:14:16.093032 20552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:14:16.092742 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.131.190:46697 (local address 127.3.131.129:59699)
0621 05:14:16.092930 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0621 05:14:16.092980 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.131.190:46697: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":164,"thread_start_us":74,"threads_started":1}
I20260621 05:14:16.161942 20553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:14:16.161683 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.131.190:46697 (local address 127.3.131.130:33789)
0621 05:14:16.161840 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0621 05:14:16.161886 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.131.190:46697: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":65,"threads_started":1}
I20260621 05:14:16.302539 20554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:14:16.302301 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.131.190:46697 (local address 127.3.131.131:52429)
0621 05:14:16.302448 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0621 05:14:16.302491 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.131.190:46697: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":65,"threads_started":1}
I20260621 05:14:16.900200 20555 raft_consensus.cc:482] T 6a2f678342444e57bb2bccae6c6af0c9 P af09019aeb194a60b128a035658ed606 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260621 05:14:16.900305 20555 raft_consensus.cc:504] T 6a2f678342444e57bb2bccae6c6af0c9 P af09019aeb194a60b128a035658ed606 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "518697964a27412aa91818fcc615e6d5" member_type: VOTER last_known_addr { host: "127.3.131.130" port: 44723 } } peers { permanent_uuid: "245dfac09a1a4fcba9d4ed15c2387fc8" member_type: VOTER last_known_addr { host: "127.3.131.131" port: 38877 } } peers { permanent_uuid: "af09019aeb194a60b128a035658ed606" member_type: VOTER last_known_addr { host: "127.3.131.129" port: 41035 } }
I20260621 05:14:16.900455 20555 leader_election.cc:288] T 6a2f678342444e57bb2bccae6c6af0c9 P af09019aeb194a60b128a035658ed606 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 518697964a27412aa91818fcc615e6d5 (127.3.131.130:44723), 245dfac09a1a4fcba9d4ed15c2387fc8 (127.3.131.131:38877)
W20260621 05:14:16.900966 20122 leader_election.cc:341] T 6a2f678342444e57bb2bccae6c6af0c9 P af09019aeb194a60b128a035658ed606 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 518697964a27412aa91818fcc615e6d5 (127.3.131.130:44723): Not found: Tablet not found: 6a2f678342444e57bb2bccae6c6af0c9
W20260621 05:14:16.901038 20122 leader_election.cc:341] T 6a2f678342444e57bb2bccae6c6af0c9 P af09019aeb194a60b128a035658ed606 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 245dfac09a1a4fcba9d4ed15c2387fc8 (127.3.131.131:38877): Not found: Tablet not found: 6a2f678342444e57bb2bccae6c6af0c9
I20260621 05:14:16.901080 20122 leader_election.cc:302] T 6a2f678342444e57bb2bccae6c6af0c9 P af09019aeb194a60b128a035658ed606 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af09019aeb194a60b128a035658ed606; no voters: 245dfac09a1a4fcba9d4ed15c2387fc8, 518697964a27412aa91818fcc615e6d5
I20260621 05:14:16.901170 20555 raft_consensus.cc:2726] T 6a2f678342444e57bb2bccae6c6af0c9 P af09019aeb194a60b128a035658ed606 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260621 05:14:17.094035 20556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:14:17.093764 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.131.190:46697 (local address 127.3.131.129:57143)
0621 05:14:17.093944 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0621 05:14:17.093990 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.131.190:46697: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":160,"thread_start_us":89,"threads_started":1}
I20260621 05:14:17.162933 20557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:14:17.162700 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.131.190:46697 (local address 127.3.131.130:53165)
0621 05:14:17.162849 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0621 05:14:17.162886 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.131.190:46697: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":105,"threads_started":1}
I20260621 05:14:17.300467 3598 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWZOx2_/build/release/bin/kudu with pid 20075
I20260621 05:14:17.303449 20558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:14:17.303239 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.131.190:46697 (local address 127.3.131.131:34083)
0621 05:14:17.303362 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0621 05:14:17.303400 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.131.190:46697: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":75,"threads_started":1}
I20260621 05:14:17.306497 3598 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWZOx2_/build/release/bin/kudu with pid 20239
I20260621 05:14:17.311911 3598 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWZOx2_/build/release/bin/kudu with pid 20373
2026-06-21T05:14:17Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (58613 ms)
[----------] 30 tests from SecurityITest (240670 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782018616648578-3598-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 21 05:14:17 dist-test-slave-vrq1 krb5kdc[20561](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 21 05:14:17 dist-test-slave-vrq1 krb5kdc[20561](info): set up 2 sockets
Jun 21 05:14:17 dist-test-slave-vrq1 krb5kdc[20561](info): commencing operation
krb5kdc: starting...
W20260621 05:14:18.379415 3598 mini_kdc.cc:121] Time spent starting KDC: real 1.046s user 0.001s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782018616648578-3598-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782018616648578-3598-0/krb5kdc/test-user.keytab.
Jun 21 05:14:18 dist-test-slave-vrq1 krb5kdc[20561](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782018858, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-21T05:14:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-21T05:14:18Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.131.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.131.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.131.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782018616648578-3598-0/krb5kdc/kudu_127.3.131.190.keytab.
Entry for principal kudu/127.3.131.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782018616648578-3598-0/krb5kdc/kudu_127.3.131.190.keytab.
I20260621 05:14:18.510653 3598 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWZOx2_/build/release/bin/kudu
/tmp/dist-test-taskWZOx2_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782018616648578-3598-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782018616648578-3598-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782018616648578-3598-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782018616648578-3598-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.131.190:40619
--webserver_interface=127.3.131.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782018616648578-3598-0/krb5kdc/kudu_127.3.131.190.keytab
--principal=kudu/127.3.131.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.131.148:46187
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.131.190:40619
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782018616648578-3598-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782018616648578-3598-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782018616648578-3598-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 05:14:18.595304 20574 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 05:14:18.595592 20574 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 05:14:18.595616 20574 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 05:14:18.596817 20574 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 05:14:18.596863 20574 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 05:14:18.596877 20574 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 05:14:18.596890 20574 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 05:14:18.596902 20574 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 05:14:18.598412 20574 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.131.148:46187
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782018616648578-3598-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782018616648578-3598-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.131.190:40619
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782018616648578-3598-0/krb5kdc/kudu_127.3.131.190.keytab
--principal=kudu/127.3.131.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.131.190:40619
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782018616648578-3598-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.131.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20574
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782018616648578-3598-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 21 Jun 2026 04:16:16 UTC on ccycloud.quasar-xycaha.root.comops.site
build id 4457881
I20260621 05:14:18.598680 20574 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 05:14:18.598918 20574 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 05:14:18.604760 20579 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:14:18.604835 20582 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:14:18.604873 20574 server_base.cc:1034] running on GCE node
W20260621 05:14:18.604835 20580 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:14:18.605297 20574 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 05:14:18.605629 20574 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 05:14:18.606834 20574 hybrid_clock.cc:648] HybridClock initialized: now 1782018858606744 us; error 98 us; skew 500 ppm
Jun 21 05:14:18 dist-test-slave-vrq1 krb5kdc[20561](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782018858, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.131.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 05:14:18.610134 20574 init.cc:377] Logged in from keytab as kudu/127.3.131.190@KRBTEST.COM (short username kudu)
I20260621 05:14:18.610874 20574 webserver.cc:460] Webserver started at http://127.3.131.190:34263/ using document root <none> and password file <none>
I20260621 05:14:18.611290 20574 fs_manager.cc:359] Metadata directory not provided
I20260621 05:14:18.611366 20574 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 05:14:18.611506 20574 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 05:14:18.612907 20574 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782018616648578-3598-0/minicluster-data/master-0/data/instance:
uuid: "ab7cef70ff3946869f3eaf8b05d0147d"
format_stamp: "Formatted at 2026-06-21 05:14:18 on dist-test-slave-vrq1"
server_key: "3e738f2f5faa9ea762db142f455ed061"
server_key_iv: "d00e9fb6eeda0895c61c46fec95c15ca"
server_key_version: "encryptionkey@0"
I20260621 05:14:18.613511 20574 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782018616648578-3598-0/minicluster-data/master-0/wal/instance:
uuid: "ab7cef70ff3946869f3eaf8b05d0147d"
format_stamp: "Formatted at 2026-06-21 05:14:18 on dist-test-slave-vrq1"
server_key: "3e738f2f5faa9ea762db142f455ed061"
server_key_iv: "d00e9fb6eeda0895c61c46fec95c15ca"
server_key_version: "encryptionkey@0"
I20260621 05:14:18.615456 20574 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.004s sys 0.000s
I20260621 05:14:18.616602 20589 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 05:14:18.617118 20574 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260621 05:14:18.617254 20574 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782018616648578-3598-0/minicluster-data/master-0/data,/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782018616648578-3598-0/minicluster-data/master-0/wal
uuid: "ab7cef70ff3946869f3eaf8b05d0147d"
format_stamp: "Formatted at 2026-06-21 05:14:18 on dist-test-slave-vrq1"
server_key: "3e738f2f5faa9ea762db142f455ed061"
server_key_iv: "d00e9fb6eeda0895c61c46fec95c15ca"
server_key_version: "encryptionkey@0"
I20260621 05:14:18.617349 20574 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782018616648578-3598-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782018616648578-3598-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782018616648578-3598-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 05:14:18.628192 20574 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 05:14:18.631644 20574 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 05:14:18.631794 20574 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 05:14:18.636448 20574 rpc_server.cc:307] RPC server started. Bound to: 127.3.131.190:40619
I20260621 05:14:18.636513 20641 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.131.190:40619 every 8 connection(s)
I20260621 05:14:18.637064 20574 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782018616648578-3598-0/minicluster-data/master-0/data/info.pb
I20260621 05:14:18.638044 20642 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 05:14:18.641095 20642 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ab7cef70ff3946869f3eaf8b05d0147d: Bootstrap starting.
I20260621 05:14:18.642086 20642 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ab7cef70ff3946869f3eaf8b05d0147d: Neither blocks nor log segments found. Creating new log.
I20260621 05:14:18.642410 20642 log.cc:824] T 00000000000000000000000000000000 P ab7cef70ff3946869f3eaf8b05d0147d: Log is configured to *not* fsync() on all Append() calls
I20260621 05:14:18.643626 20642 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ab7cef70ff3946869f3eaf8b05d0147d: No bootstrap required, opened a new log
I20260621 05:14:18.645376 20642 raft_consensus.cc:348] T 00000000000000000000000000000000 P ab7cef70ff3946869f3eaf8b05d0147d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ab7cef70ff3946869f3eaf8b05d0147d" member_type: VOTER last_known_addr { host: "127.3.131.190" port: 40619 } }
I20260621 05:14:18.645553 20642 raft_consensus.cc:374] T 00000000000000000000000000000000 P ab7cef70ff3946869f3eaf8b05d0147d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 05:14:18.645578 20642 raft_consensus.cc:729] T 00000000000000000000000000000000 P ab7cef70ff3946869f3eaf8b05d0147d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ab7cef70ff3946869f3eaf8b05d0147d, State: Initialized, Role: FOLLOWER
I20260621 05:14:18.645754 20642 consensus_queue.cc:260] T 00000000000000000000000000000000 P ab7cef70ff3946869f3eaf8b05d0147d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ab7cef70ff3946869f3eaf8b05d0147d" member_type: VOTER last_known_addr { host: "127.3.131.190" port: 40619 } }
I20260621 05:14:18.645838 20642 raft_consensus.cc:388] T 00000000000000000000000000000000 P ab7cef70ff3946869f3eaf8b05d0147d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260621 05:14:18.645862 20642 raft_consensus.cc:482] T 00000000000000000000000000000000 P ab7cef70ff3946869f3eaf8b05d0147d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260621 05:14:18.645893 20642 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ab7cef70ff3946869f3eaf8b05d0147d [term 0 FOLLOWER]: Advancing to term 1
I20260621 05:14:18.646509 3598 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWZOx2_/build/release/bin/kudu as pid 20574
I20260621 05:14:18.646629 3598 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782018616648578-3598-0/minicluster-data/master-0/wal/instance
I20260621 05:14:18.646821 3598 external_mini_cluster.cc:1428] Setting key 1459a5057580b48d48f13e056f74fa4b
I20260621 05:14:18.647059 20642 raft_consensus.cc:504] T 00000000000000000000000000000000 P ab7cef70ff3946869f3eaf8b05d0147d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ab7cef70ff3946869f3eaf8b05d0147d" member_type: VOTER last_known_addr { host: "127.3.131.190" port: 40619 } }
I20260621 05:14:18.647210 20642 leader_election.cc:302] T 00000000000000000000000000000000 P ab7cef70ff3946869f3eaf8b05d0147d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ab7cef70ff3946869f3eaf8b05d0147d; no voters:
I20260621 05:14:18.647409 20642 leader_election.cc:288] T 00000000000000000000000000000000 P ab7cef70ff3946869f3eaf8b05d0147d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260621 05:14:18.647521 20645 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ab7cef70ff3946869f3eaf8b05d0147d [term 1 FOLLOWER]: Leader election won for term 1
I20260621 05:14:18.647701 20645 raft_consensus.cc:686] T 00000000000000000000000000000000 P ab7cef70ff3946869f3eaf8b05d0147d [term 1 LEADER]: Becoming Leader. State: Replica: ab7cef70ff3946869f3eaf8b05d0147d, State: Running, Role: LEADER
I20260621 05:14:18.647825 20645 consensus_queue.cc:237] T 00000000000000000000000000000000 P ab7cef70ff3946869f3eaf8b05d0147d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ab7cef70ff3946869f3eaf8b05d0147d" member_type: VOTER last_known_addr { host: "127.3.131.190" port: 40619 } }
I20260621 05:14:18.647864 20642 sys_catalog.cc:564] T 00000000000000000000000000000000 P ab7cef70ff3946869f3eaf8b05d0147d [sys.catalog]: configured and running, proceeding with master startup.
I20260621 05:14:18.648186 20645 sys_catalog.cc:455] T 00000000000000000000000000000000 P ab7cef70ff3946869f3eaf8b05d0147d [sys.catalog]: SysCatalogTable state changed. Reason: New leader ab7cef70ff3946869f3eaf8b05d0147d. Latest consensus state: current_term: 1 leader_uuid: "ab7cef70ff3946869f3eaf8b05d0147d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ab7cef70ff3946869f3eaf8b05d0147d" member_type: VOTER last_known_addr { host: "127.3.131.190" port: 40619 } } }
I20260621 05:14:18.648293 20645 sys_catalog.cc:458] T 00000000000000000000000000000000 P ab7cef70ff3946869f3eaf8b05d0147d [sys.catalog]: This master's current role is: LEADER
I20260621 05:14:18.648612 20652 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260621 05:14:18.649205 20652 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260621 05:14:18.649739 20648 sys_catalog.cc:455] T 00000000000000000000000000000000 P ab7cef70ff3946869f3eaf8b05d0147d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ab7cef70ff3946869f3eaf8b05d0147d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ab7cef70ff3946869f3eaf8b05d0147d" member_type: VOTER last_known_addr { host: "127.3.131.190" port: 40619 } } }
I20260621 05:14:18.649825 20648 sys_catalog.cc:458] T 00000000000000000000000000000000 P ab7cef70ff3946869f3eaf8b05d0147d [sys.catalog]: This master's current role is: LEADER
I20260621 05:14:18.650701 20652 catalog_manager.cc:1347] Generated new cluster ID: 6be19a8728b14e778065a45275af5122
I20260621 05:14:18.650758 20652 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Jun 21 05:14:18 dist-test-slave-vrq1 krb5kdc[20561](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782018858, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.131.190@KRBTEST.COM
I20260621 05:14:18.657862 20647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:14:18.647508 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45954 (local address 127.3.131.190:40619)
0621 05:14:18.647782 (+ 274us) server_negotiation.cc:206] Beginning negotiation
0621 05:14:18.647793 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0621 05:14:18.647826 (+ 33us) server_negotiation.cc:407] Connection header received
0621 05:14:18.647956 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:14:18.647964 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:14:18.648026 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:14:18.648192 (+ 166us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:14:18.649099 (+ 907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:14:18.650503 (+ 1404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:14:18.651477 (+ 974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:14:18.651914 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:14:18.652038 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:14:18.654435 (+ 2397us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:14:18.654440 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:14:18.654440 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:14:18.654451 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:14:18.656228 (+ 1777us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:14:18.656612 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:14:18.656616 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:14:18.656616 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:14:18.656727 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:14:18.656935 (+ 208us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:14:18.656936 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:14:18.656937 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:14:18.657202 (+ 265us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:14:18.657384 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0621 05:14:18.657452 (+ 68us) server_negotiation.cc:299] Negotiation successful
0621 05:14:18.657702 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":216,"thread_start_us":113,"threads_started":1}
I20260621 05:14:18.670370 20652 catalog_manager.cc:1370] Generated new certificate authority record
I20260621 05:14:18.671144 20652 catalog_manager.cc:1504] Loading token signing keys...
I20260621 05:14:18.675374 20652 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ab7cef70ff3946869f3eaf8b05d0147d: Generated new TSK 0
I20260621 05:14:18.675755 20652 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.3.131.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.131.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.131.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782018616648578-3598-0/krb5kdc/kudu_127.3.131.129.keytab.
Entry for principal kudu/127.3.131.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782018616648578-3598-0/krb5kdc/kudu_127.3.131.129.keytab.
I20260621 05:14:18.712626 3598 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWZOx2_/build/release/bin/kudu
/tmp/dist-test-taskWZOx2_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782018616648578-3598-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782018616648578-3598-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782018616648578-3598-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782018616648578-3598-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.131.129:0
--local_ip_for_outbound_sockets=127.3.131.129
--webserver_interface=127.3.131.129
--webserver_port=0
--tserver_master_addrs=127.3.131.190:40619
--keytab_file=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782018616648578-3598-0/krb5kdc/kudu_127.3.131.129.keytab
--principal=kudu/127.3.131.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.131.148:46187
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782018616648578-3598-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782018616648578-3598-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782018616648578-3598-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 05:14:18.814010 20668 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 05:14:18.814240 20668 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 05:14:18.814268 20668 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 05:14:18.815697 20668 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 05:14:18.815783 20668 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 05:14:18.815815 20668 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.131.129
I20260621 05:14:18.817622 20668 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.131.148:46187
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782018616648578-3598-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782018616648578-3598-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782018616648578-3598-0/krb5kdc/kudu_127.3.131.129.keytab
--principal=kudu/127.3.131.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.131.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782018616648578-3598-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.131.129
--webserver_port=0
--tserver_master_addrs=127.3.131.190:40619
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20668
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.131.129
--log_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782018616648578-3598-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 21 Jun 2026 04:16:16 UTC on ccycloud.quasar-xycaha.root.comops.site
build id 4457881
I20260621 05:14:18.817906 20668 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 05:14:18.818192 20668 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 05:14:18.823869 20676 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:14:18.823899 20673 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:14:18.823869 20674 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:14:18.824235 20668 server_base.cc:1034] running on GCE node
I20260621 05:14:18.824420 20668 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 05:14:18.824716 20668 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 05:14:18.825911 20668 hybrid_clock.cc:648] HybridClock initialized: now 1782018858825897 us; error 31 us; skew 500 ppm
Jun 21 05:14:18 dist-test-slave-vrq1 krb5kdc[20561](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782018858, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.131.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 05:14:18.829136 20668 init.cc:377] Logged in from keytab as kudu/127.3.131.129@KRBTEST.COM (short username kudu)
I20260621 05:14:18.829664 20668 webserver.cc:460] Webserver started at http://127.3.131.129:33909/ using document root <none> and password file <none>
I20260621 05:14:18.829924 20668 fs_manager.cc:359] Metadata directory not provided
I20260621 05:14:18.829995 20668 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 05:14:18.830152 20668 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 05:14:18.831817 20668 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782018616648578-3598-0/minicluster-data/ts-0/data/instance:
uuid: "78a242bbee3d4eea8ce334817dfdfe5c"
format_stamp: "Formatted at 2026-06-21 05:14:18 on dist-test-slave-vrq1"
server_key: "c2a8cd26ec5eb00cdd9fc1246f0b96b9"
server_key_iv: "f4de86b067778dc7bf32107062300377"
server_key_version: "encryptionkey@0"
I20260621 05:14:18.832250 20668 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782018616648578-3598-0/minicluster-data/ts-0/wal/instance:
uuid: "78a242bbee3d4eea8ce334817dfdfe5c"
format_stamp: "Formatted at 2026-06-21 05:14:18 on dist-test-slave-vrq1"
server_key: "c2a8cd26ec5eb00cdd9fc1246f0b96b9"
server_key_iv: "f4de86b067778dc7bf32107062300377"
server_key_version: "encryptionkey@0"
I20260621 05:14:18.833835 20668 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260621 05:14:18.834822 20683 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 05:14:18.835151 20668 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260621 05:14:18.835347 20668 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782018616648578-3598-0/minicluster-data/ts-0/data,/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782018616648578-3598-0/minicluster-data/ts-0/wal
uuid: "78a242bbee3d4eea8ce334817dfdfe5c"
format_stamp: "Formatted at 2026-06-21 05:14:18 on dist-test-slave-vrq1"
server_key: "c2a8cd26ec5eb00cdd9fc1246f0b96b9"
server_key_iv: "f4de86b067778dc7bf32107062300377"
server_key_version: "encryptionkey@0"
I20260621 05:14:18.835434 20668 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782018616648578-3598-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782018616648578-3598-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782018616648578-3598-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 05:14:18.850170 20668 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 05:14:18.852864 20668 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 05:14:18.853058 20668 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 05:14:18.853320 20668 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 05:14:18.853662 20668 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 05:14:18.853724 20668 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 05:14:18.853761 20668 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 05:14:18.853803 20668 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 05:14:18.860069 20668 rpc_server.cc:307] RPC server started. Bound to: 127.3.131.129:40473
I20260621 05:14:18.860155 20796 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.131.129:40473 every 8 connection(s)
I20260621 05:14:18.860495 20668 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782018616648578-3598-0/minicluster-data/ts-0/data/info.pb
Jun 21 05:14:18 dist-test-slave-vrq1 krb5kdc[20561](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782018858, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.131.129@KRBTEST.COM for kudu/127.3.131.190@KRBTEST.COM
I20260621 05:14:18.869686 3598 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWZOx2_/build/release/bin/kudu as pid 20668
I20260621 05:14:18.869980 3598 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782018616648578-3598-0/minicluster-data/ts-0/wal/instance
I20260621 05:14:18.870280 3598 external_mini_cluster.cc:1428] Setting key e882e70cc6749a26f7b5eb0e4521bc93
I20260621 05:14:18.872439 20647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:14:18.861341 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.131.129:58347 (local address 127.3.131.190:40619)
0621 05:14:18.861421 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0621 05:14:18.861424 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0621 05:14:18.861673 (+ 249us) server_negotiation.cc:407] Connection header received
0621 05:14:18.862214 (+ 541us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:14:18.862215 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:14:18.862231 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:14:18.862296 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:14:18.863583 (+ 1287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:14:18.864127 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:14:18.865321 (+ 1194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:14:18.865527 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:14:18.865594 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:14:18.868109 (+ 2515us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:14:18.868110 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:14:18.868111 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:14:18.868118 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:14:18.870442 (+ 2324us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:14:18.871078 (+ 636us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:14:18.871080 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:14:18.871081 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:14:18.871228 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:14:18.871551 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:14:18.871552 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:14:18.871553 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:14:18.871721 (+ 168us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:14:18.871804 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0621 05:14:18.872191 (+ 387us) server_negotiation.cc:299] Negotiation successful
0621 05:14:18.872294 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260621 05:14:18.872748 20799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:14:18.861287 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.131.190:40619 (local address 127.3.131.129:58347)
0621 05:14:18.861517 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0621 05:14:18.861558 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0621 05:14:18.862059 (+ 501us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:14:18.862472 (+ 413us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:14:18.862477 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:14:18.863112 (+ 635us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:14:18.863427 (+ 315us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:14:18.863433 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:14:18.864244 (+ 811us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:14:18.864245 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:14:18.865222 (+ 977us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:14:18.865226 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:14:18.865633 (+ 407us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:14:18.865633 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:14:18.865811 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:14:18.866393 (+ 582us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 05:14:18.866402 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:14:18.867997 (+ 1595us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 05:14:18.870550 (+ 2553us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:14:18.870563 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:14:18.870566 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:14:18.870977 (+ 411us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:14:18.871323 (+ 346us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:14:18.871325 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:14:18.871325 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:14:18.871476 (+ 151us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:14:18.871826 (+ 350us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 05:14:18.871834 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 05:14:18.871983 (+ 149us) client_negotiation.cc:769] Sending connection context
0621 05:14:18.872170 (+ 187us) client_negotiation.cc:240] Negotiation successful
0621 05:14:18.872445 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":175,"thread_start_us":118,"threads_started":1}
I20260621 05:14:18.873390 20797 heartbeater.cc:347] Connected to a master server at 127.3.131.190:40619
I20260621 05:14:18.873519 20797 heartbeater.cc:464] Registering TS with master...
I20260621 05:14:18.873759 20797 heartbeater.cc:511] Master 127.3.131.190:40619 requested a full tablet report, sending...
I20260621 05:14:18.874449 20606 ts_manager.cc:194] Registered new tserver with Master: 78a242bbee3d4eea8ce334817dfdfe5c (127.3.131.129:40473)
I20260621 05:14:18.875092 20606 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.131.129@KRBTEST.COM'} at 127.3.131.129:58347
WARNING: no policy specified for kudu/127.3.131.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.131.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.131.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782018616648578-3598-0/krb5kdc/kudu_127.3.131.130.keytab.
Entry for principal kudu/127.3.131.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782018616648578-3598-0/krb5kdc/kudu_127.3.131.130.keytab.
I20260621 05:14:18.910427 3598 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWZOx2_/build/release/bin/kudu
/tmp/dist-test-taskWZOx2_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782018616648578-3598-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782018616648578-3598-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782018616648578-3598-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782018616648578-3598-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.131.130:0
--local_ip_for_outbound_sockets=127.3.131.130
--webserver_interface=127.3.131.130
--webserver_port=0
--tserver_master_addrs=127.3.131.190:40619
--keytab_file=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782018616648578-3598-0/krb5kdc/kudu_127.3.131.130.keytab
--principal=kudu/127.3.131.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.131.148:46187
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782018616648578-3598-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782018616648578-3598-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782018616648578-3598-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 05:14:18.997431 20802 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 05:14:18.997711 20802 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 05:14:18.997766 20802 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 05:14:18.999013 20802 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 05:14:18.999095 20802 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 05:14:18.999142 20802 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.131.130
I20260621 05:14:19.000836 20802 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.131.148:46187
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782018616648578-3598-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782018616648578-3598-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782018616648578-3598-0/krb5kdc/kudu_127.3.131.130.keytab
--principal=kudu/127.3.131.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.131.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782018616648578-3598-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.131.130
--webserver_port=0
--tserver_master_addrs=127.3.131.190:40619
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20802
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.131.130
--log_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782018616648578-3598-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 21 Jun 2026 04:16:16 UTC on ccycloud.quasar-xycaha.root.comops.site
build id 4457881
I20260621 05:14:19.001124 20802 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 05:14:19.001356 20802 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 05:14:19.007308 20807 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:14:19.007283 20810 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:14:19.007283 20808 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:14:19.007706 20802 server_base.cc:1034] running on GCE node
I20260621 05:14:19.007931 20802 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 05:14:19.008214 20802 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 05:14:19.009450 20802 hybrid_clock.cc:648] HybridClock initialized: now 1782018859009411 us; error 52 us; skew 500 ppm
Jun 21 05:14:19 dist-test-slave-vrq1 krb5kdc[20561](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782018859, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.131.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 05:14:19.012605 20802 init.cc:377] Logged in from keytab as kudu/127.3.131.130@KRBTEST.COM (short username kudu)
I20260621 05:14:19.013077 20802 webserver.cc:460] Webserver started at http://127.3.131.130:40001/ using document root <none> and password file <none>
I20260621 05:14:19.013334 20802 fs_manager.cc:359] Metadata directory not provided
I20260621 05:14:19.013402 20802 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 05:14:19.013545 20802 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 05:14:19.014633 20802 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782018616648578-3598-0/minicluster-data/ts-1/data/instance:
uuid: "ed2350f926634e20b250a0c82a62251c"
format_stamp: "Formatted at 2026-06-21 05:14:19 on dist-test-slave-vrq1"
server_key: "ab146164af3d925da2ca166a621b3188"
server_key_iv: "55277fe58019901bce17d1eaa8d2c9eb"
server_key_version: "encryptionkey@0"
I20260621 05:14:19.015004 20802 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782018616648578-3598-0/minicluster-data/ts-1/wal/instance:
uuid: "ed2350f926634e20b250a0c82a62251c"
format_stamp: "Formatted at 2026-06-21 05:14:19 on dist-test-slave-vrq1"
server_key: "ab146164af3d925da2ca166a621b3188"
server_key_iv: "55277fe58019901bce17d1eaa8d2c9eb"
server_key_version: "encryptionkey@0"
I20260621 05:14:19.016578 20802 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260621 05:14:19.017524 20817 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 05:14:19.017812 20802 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260621 05:14:19.017925 20802 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782018616648578-3598-0/minicluster-data/ts-1/data,/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782018616648578-3598-0/minicluster-data/ts-1/wal
uuid: "ed2350f926634e20b250a0c82a62251c"
format_stamp: "Formatted at 2026-06-21 05:14:19 on dist-test-slave-vrq1"
server_key: "ab146164af3d925da2ca166a621b3188"
server_key_iv: "55277fe58019901bce17d1eaa8d2c9eb"
server_key_version: "encryptionkey@0"
I20260621 05:14:19.018033 20802 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782018616648578-3598-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782018616648578-3598-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782018616648578-3598-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 05:14:19.028297 20802 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 05:14:19.030872 20802 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 05:14:19.031057 20802 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 05:14:19.031368 20802 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 05:14:19.031699 20802 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 05:14:19.031754 20802 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 05:14:19.031797 20802 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 05:14:19.031827 20802 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 05:14:19.038312 20802 rpc_server.cc:307] RPC server started. Bound to: 127.3.131.130:42669
I20260621 05:14:19.038395 20930 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.131.130:42669 every 8 connection(s)
I20260621 05:14:19.038769 20802 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782018616648578-3598-0/minicluster-data/ts-1/data/info.pb
Jun 21 05:14:19 dist-test-slave-vrq1 krb5kdc[20561](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782018859, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.131.130@KRBTEST.COM for kudu/127.3.131.190@KRBTEST.COM
I20260621 05:14:19.046162 3598 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWZOx2_/build/release/bin/kudu as pid 20802
I20260621 05:14:19.046267 3598 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782018616648578-3598-0/minicluster-data/ts-1/wal/instance
I20260621 05:14:19.046459 3598 external_mini_cluster.cc:1428] Setting key 813e4b4e8517b87788e03c4048311ba2
I20260621 05:14:19.049742 20647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:14:19.039773 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.131.130:38101 (local address 127.3.131.190:40619)
0621 05:14:19.039837 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0621 05:14:19.039839 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:14:19.040023 (+ 184us) server_negotiation.cc:407] Connection header received
0621 05:14:19.040500 (+ 477us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:14:19.040502 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:14:19.040513 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:14:19.040580 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:14:19.041610 (+ 1030us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:14:19.042133 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:14:19.043296 (+ 1163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:14:19.043503 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:14:19.043569 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:14:19.045949 (+ 2380us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:14:19.045951 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:14:19.045951 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:14:19.045961 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:14:19.047874 (+ 1913us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:14:19.048498 (+ 624us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:14:19.048500 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:14:19.048500 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:14:19.048571 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:14:19.048875 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:14:19.048876 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:14:19.048877 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:14:19.049055 (+ 178us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:14:19.049177 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0621 05:14:19.049515 (+ 338us) server_negotiation.cc:299] Negotiation successful
0621 05:14:19.049630 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260621 05:14:19.049829 20933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:14:19.039694 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.131.190:40619 (local address 127.3.131.130:38101)
0621 05:14:19.039870 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0621 05:14:19.039913 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0621 05:14:19.040395 (+ 482us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:14:19.040687 (+ 292us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:14:19.040691 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:14:19.041296 (+ 605us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:14:19.041498 (+ 202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:14:19.041503 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:14:19.042277 (+ 774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:14:19.042278 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:14:19.043135 (+ 857us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:14:19.043138 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:14:19.043585 (+ 447us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:14:19.043586 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:14:19.043708 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:14:19.044264 (+ 556us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 05:14:19.044273 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:14:19.045821 (+ 1548us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 05:14:19.047969 (+ 2148us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:14:19.047978 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:14:19.047981 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:14:19.048406 (+ 425us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:14:19.048642 (+ 236us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:14:19.048643 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:14:19.048643 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:14:19.048798 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:14:19.049146 (+ 348us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 05:14:19.049153 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 05:14:19.049376 (+ 223us) client_negotiation.cc:769] Sending connection context
0621 05:14:19.049472 (+ 96us) client_negotiation.cc:240] Negotiation successful
0621 05:14:19.049613 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":96,"threads_started":1}
I20260621 05:14:19.050355 20931 heartbeater.cc:347] Connected to a master server at 127.3.131.190:40619
I20260621 05:14:19.050462 20931 heartbeater.cc:464] Registering TS with master...
I20260621 05:14:19.050659 20931 heartbeater.cc:511] Master 127.3.131.190:40619 requested a full tablet report, sending...
I20260621 05:14:19.051082 20606 ts_manager.cc:194] Registered new tserver with Master: ed2350f926634e20b250a0c82a62251c (127.3.131.130:42669)
I20260621 05:14:19.051501 20606 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.131.130@KRBTEST.COM'} at 127.3.131.130:38101
WARNING: no policy specified for kudu/127.3.131.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.131.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.131.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782018616648578-3598-0/krb5kdc/kudu_127.3.131.131.keytab.
Entry for principal kudu/127.3.131.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782018616648578-3598-0/krb5kdc/kudu_127.3.131.131.keytab.
I20260621 05:14:19.074353 3598 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWZOx2_/build/release/bin/kudu
/tmp/dist-test-taskWZOx2_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782018616648578-3598-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782018616648578-3598-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782018616648578-3598-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782018616648578-3598-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.131.131:0
--local_ip_for_outbound_sockets=127.3.131.131
--webserver_interface=127.3.131.131
--webserver_port=0
--tserver_master_addrs=127.3.131.190:40619
--keytab_file=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782018616648578-3598-0/krb5kdc/kudu_127.3.131.131.keytab
--principal=kudu/127.3.131.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.131.148:46187
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782018616648578-3598-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782018616648578-3598-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782018616648578-3598-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 05:14:19.159155 20936 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 05:14:19.159449 20936 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 05:14:19.159474 20936 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 05:14:19.160686 20936 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 05:14:19.160729 20936 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 05:14:19.160756 20936 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.131.131
I20260621 05:14:19.162189 20936 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.131.148:46187
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782018616648578-3598-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782018616648578-3598-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782018616648578-3598-0/krb5kdc/kudu_127.3.131.131.keytab
--principal=kudu/127.3.131.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.131.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782018616648578-3598-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.131.131
--webserver_port=0
--tserver_master_addrs=127.3.131.190:40619
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20936
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.131.131
--log_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782018616648578-3598-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 21 Jun 2026 04:16:16 UTC on ccycloud.quasar-xycaha.root.comops.site
build id 4457881
I20260621 05:14:19.162391 20936 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 05:14:19.162575 20936 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 05:14:19.168040 20944 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:14:19.168042 20941 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:14:19.168538 20942 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:14:19.168552 20936 server_base.cc:1034] running on GCE node
I20260621 05:14:19.168871 20936 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 05:14:19.169100 20936 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 05:14:19.170286 20936 hybrid_clock.cc:648] HybridClock initialized: now 1782018859170270 us; error 32 us; skew 500 ppm
Jun 21 05:14:19 dist-test-slave-vrq1 krb5kdc[20561](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782018859, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.131.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 05:14:19.173451 20936 init.cc:377] Logged in from keytab as kudu/127.3.131.131@KRBTEST.COM (short username kudu)
I20260621 05:14:19.174381 20936 webserver.cc:460] Webserver started at http://127.3.131.131:40513/ using document root <none> and password file <none>
I20260621 05:14:19.174825 20936 fs_manager.cc:359] Metadata directory not provided
I20260621 05:14:19.174889 20936 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 05:14:19.175026 20936 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 05:14:19.176466 20936 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782018616648578-3598-0/minicluster-data/ts-2/data/instance:
uuid: "745736f5f556409fa20a260fb647ab93"
format_stamp: "Formatted at 2026-06-21 05:14:19 on dist-test-slave-vrq1"
server_key: "be1487e84112ae3f236bcf85ec715eca"
server_key_iv: "a10130d822bb3f01f4cf507fb778de13"
server_key_version: "encryptionkey@0"
I20260621 05:14:19.176944 20936 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782018616648578-3598-0/minicluster-data/ts-2/wal/instance:
uuid: "745736f5f556409fa20a260fb647ab93"
format_stamp: "Formatted at 2026-06-21 05:14:19 on dist-test-slave-vrq1"
server_key: "be1487e84112ae3f236bcf85ec715eca"
server_key_iv: "a10130d822bb3f01f4cf507fb778de13"
server_key_version: "encryptionkey@0"
I20260621 05:14:19.178505 20936 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260621 05:14:19.179534 20951 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 05:14:19.179836 20936 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260621 05:14:19.179925 20936 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782018616648578-3598-0/minicluster-data/ts-2/data,/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782018616648578-3598-0/minicluster-data/ts-2/wal
uuid: "745736f5f556409fa20a260fb647ab93"
format_stamp: "Formatted at 2026-06-21 05:14:19 on dist-test-slave-vrq1"
server_key: "be1487e84112ae3f236bcf85ec715eca"
server_key_iv: "a10130d822bb3f01f4cf507fb778de13"
server_key_version: "encryptionkey@0"
I20260621 05:14:19.179991 20936 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782018616648578-3598-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782018616648578-3598-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782018616648578-3598-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 05:14:19.194732 20936 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 05:14:19.197033 20936 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 05:14:19.197158 20936 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 05:14:19.197360 20936 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 05:14:19.197698 20936 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 05:14:19.197732 20936 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 05:14:19.197778 20936 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 05:14:19.197803 20936 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 05:14:19.204154 20936 rpc_server.cc:307] RPC server started. Bound to: 127.3.131.131:42983
I20260621 05:14:19.204209 21064 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.131.131:42983 every 8 connection(s)
I20260621 05:14:19.204524 20936 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782018616648578-3598-0/minicluster-data/ts-2/data/info.pb
I20260621 05:14:19.210212 3598 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWZOx2_/build/release/bin/kudu as pid 20936
I20260621 05:14:19.210305 3598 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782018616648578-3598-0/minicluster-data/ts-2/wal/instance
I20260621 05:14:19.210490 3598 external_mini_cluster.cc:1428] Setting key 943eadc26b3884150941e5afc65b74e0
Jun 21 05:14:19 dist-test-slave-vrq1 krb5kdc[20561](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782018859, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.131.131@KRBTEST.COM for kudu/127.3.131.190@KRBTEST.COM
I20260621 05:14:19.214491 20647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:14:19.205240 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.131.131:47529 (local address 127.3.131.190:40619)
0621 05:14:19.205302 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0621 05:14:19.205305 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0621 05:14:19.205517 (+ 212us) server_negotiation.cc:407] Connection header received
0621 05:14:19.205992 (+ 475us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:14:19.205993 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:14:19.206005 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:14:19.206076 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:14:19.206996 (+ 920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:14:19.207629 (+ 633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:14:19.208661 (+ 1032us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:14:19.208850 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:14:19.208913 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:14:19.211469 (+ 2556us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:14:19.211471 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:14:19.211471 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:14:19.211479 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:14:19.212999 (+ 1520us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:14:19.213491 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:14:19.213492 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:14:19.213493 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:14:19.213555 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:14:19.213819 (+ 264us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:14:19.213820 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:14:19.213820 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:14:19.213943 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:14:19.214017 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0621 05:14:19.214310 (+ 293us) server_negotiation.cc:299] Negotiation successful
0621 05:14:19.214393 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260621 05:14:19.214711 21067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:14:19.205208 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.131.190:40619 (local address 127.3.131.131:47529)
0621 05:14:19.205367 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0621 05:14:19.205410 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0621 05:14:19.205897 (+ 487us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:14:19.206133 (+ 236us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:14:19.206136 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:14:19.206706 (+ 570us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:14:19.206902 (+ 196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:14:19.206906 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:14:19.207748 (+ 842us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:14:19.207749 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:14:19.208571 (+ 822us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:14:19.208573 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:14:19.208932 (+ 359us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:14:19.208933 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:14:19.209042 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:14:19.209549 (+ 507us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 05:14:19.209557 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:14:19.211358 (+ 1801us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 05:14:19.213106 (+ 1748us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:14:19.213112 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:14:19.213114 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:14:19.213395 (+ 281us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:14:19.213632 (+ 237us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:14:19.213633 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:14:19.213633 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:14:19.213749 (+ 116us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:14:19.214005 (+ 256us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 05:14:19.214012 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 05:14:19.214178 (+ 166us) client_negotiation.cc:769] Sending connection context
0621 05:14:19.214287 (+ 109us) client_negotiation.cc:240] Negotiation successful
0621 05:14:19.214456 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":89,"threads_started":1}
I20260621 05:14:19.215232 21065 heartbeater.cc:347] Connected to a master server at 127.3.131.190:40619
I20260621 05:14:19.215358 21065 heartbeater.cc:464] Registering TS with master...
I20260621 05:14:19.215545 21065 heartbeater.cc:511] Master 127.3.131.190:40619 requested a full tablet report, sending...
I20260621 05:14:19.215929 20606 ts_manager.cc:194] Registered new tserver with Master: 745736f5f556409fa20a260fb647ab93 (127.3.131.131:42983)
I20260621 05:14:19.216331 20606 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.131.131@KRBTEST.COM'} at 127.3.131.131:47529
I20260621 05:14:19.222922 3598 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260621 05:14:19.231355 20647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:14:19.224392 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45962 (local address 127.3.131.190:40619)
0621 05:14:19.224467 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0621 05:14:19.224469 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:14:19.224513 (+ 44us) server_negotiation.cc:407] Connection header received
0621 05:14:19.224646 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:14:19.224648 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:14:19.224659 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:14:19.224831 (+ 172us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:14:19.225546 (+ 715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:14:19.226071 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:14:19.226873 (+ 802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:14:19.227074 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:14:19.227149 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:14:19.228485 (+ 1336us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:14:19.228486 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:14:19.228487 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:14:19.228493 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:14:19.229967 (+ 1474us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:14:19.230370 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:14:19.230371 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:14:19.230372 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:14:19.230438 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:14:19.230743 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:14:19.230744 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:14:19.230745 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:14:19.230916 (+ 171us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:14:19.231029 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0621 05:14:19.231133 (+ 104us) server_negotiation.cc:299] Negotiation successful
0621 05:14:19.231251 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260621 05:14:19.233413 20606 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45962:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260621 05:14:19.235134 20606 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260621 05:14:19.242691 21077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:14:19.238060 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.131.129:40473 (local address 127.0.0.1:35076)
0621 05:14:19.238419 (+ 359us) negotiation.cc:107] Waiting for socket to connect
0621 05:14:19.238428 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0621 05:14:19.238468 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:14:19.239041 (+ 573us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:14:19.239042 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:14:19.239045 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 05:14:19.239105 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:14:19.239107 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:14:19.240488 (+ 1381us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:14:19.240489 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:14:19.241328 (+ 839us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:14:19.241331 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:14:19.242307 (+ 976us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:14:19.242309 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:14:19.242402 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:14:19.242418 (+ 16us) client_negotiation.cc:769] Sending connection context
0621 05:14:19.242510 (+ 92us) client_negotiation.cc:240] Negotiation successful
0621 05:14:19.242574 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":342,"thread_start_us":86,"threads_started":1}
I20260621 05:14:19.242697 21078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:14:19.238196 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35076 (local address 127.3.131.129:40473)
0621 05:14:19.238784 (+ 588us) server_negotiation.cc:206] Beginning negotiation
0621 05:14:19.238788 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 05:14:19.238797 (+ 9us) server_negotiation.cc:407] Connection header received
0621 05:14:19.238871 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:14:19.238873 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:14:19.238891 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:14:19.238978 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 05:14:19.239300 (+ 322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:14:19.240384 (+ 1084us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:14:19.241417 (+ 1033us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:14:19.242197 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:14:19.242275 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:14:19.242336 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0621 05:14:19.242535 (+ 199us) server_negotiation.cc:299] Negotiation successful
0621 05:14:19.242586 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":554,"thread_start_us":111,"threads_started":1}
I20260621 05:14:19.242971 21076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:14:19.238050 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.131.131:42983 (local address 127.0.0.1:49442)
0621 05:14:19.238179 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0621 05:14:19.238225 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0621 05:14:19.238298 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:14:19.238802 (+ 504us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:14:19.238805 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:14:19.238807 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 05:14:19.238885 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:14:19.238887 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:14:19.240169 (+ 1282us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:14:19.240170 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:14:19.241666 (+ 1496us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:14:19.241670 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:14:19.242669 (+ 999us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:14:19.242670 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:14:19.242753 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:14:19.242758 (+ 5us) client_negotiation.cc:769] Sending connection context
0621 05:14:19.242796 (+ 38us) client_negotiation.cc:240] Negotiation successful
0621 05:14:19.242858 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":99,"mutex_wait_us":60,"thread_start_us":66,"threads_started":1}
I20260621 05:14:19.243292 21081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:14:19.238581 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58416 (local address 127.3.131.130:42669)
0621 05:14:19.238768 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0621 05:14:19.238772 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 05:14:19.238867 (+ 95us) server_negotiation.cc:407] Connection header received
0621 05:14:19.238908 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:14:19.238910 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:14:19.238926 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:14:19.238994 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 05:14:19.239212 (+ 218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:14:19.240352 (+ 1140us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:14:19.241849 (+ 1497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:14:19.242570 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:14:19.242637 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:14:19.242689 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0621 05:14:19.243087 (+ 398us) server_negotiation.cc:299] Negotiation successful
0621 05:14:19.243152 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":151,"thread_start_us":68,"threads_started":1}
I20260621 05:14:19.243312 21080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:14:19.238410 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49442 (local address 127.3.131.131:42983)
0621 05:14:19.238616 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0621 05:14:19.238621 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 05:14:19.238633 (+ 12us) server_negotiation.cc:407] Connection header received
0621 05:14:19.238677 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:14:19.238679 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:14:19.238695 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:14:19.238796 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 05:14:19.238957 (+ 161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:14:19.240027 (+ 1070us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:14:19.241809 (+ 1782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:14:19.242567 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:14:19.243015 (+ 448us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:14:19.243091 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0621 05:14:19.243125 (+ 34us) server_negotiation.cc:299] Negotiation successful
0621 05:14:19.243215 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":161,"thread_start_us":91,"threads_started":1}
I20260621 05:14:19.243440 21079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:14:19.238276 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.131.130:42669 (local address 127.0.0.1:58416)
0621 05:14:19.238562 (+ 286us) negotiation.cc:107] Waiting for socket to connect
0621 05:14:19.238570 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0621 05:14:19.238607 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:14:19.239067 (+ 460us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:14:19.239067 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:14:19.239069 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 05:14:19.239104 (+ 35us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:14:19.239107 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:14:19.240461 (+ 1354us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:14:19.240462 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:14:19.241759 (+ 1297us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:14:19.241761 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:14:19.242730 (+ 969us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:14:19.242731 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:14:19.242799 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:14:19.242804 (+ 5us) client_negotiation.cc:769] Sending connection context
0621 05:14:19.242855 (+ 51us) client_negotiation.cc:240] Negotiation successful
0621 05:14:19.242901 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":260,"mutex_wait_us":32,"thread_start_us":76,"threads_started":1}
I20260621 05:14:19.243806 20731 tablet_service.cc:1467] Processing CreateTablet for tablet b5ceea71b55a496bb2947def2bc4a9ae (DEFAULT_TABLE table=test-table [id=79973975394b4b0193859526d0a1bc37]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 05:14:19.243997 20999 tablet_service.cc:1467] Processing CreateTablet for tablet b5ceea71b55a496bb2947def2bc4a9ae (DEFAULT_TABLE table=test-table [id=79973975394b4b0193859526d0a1bc37]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 05:14:19.244187 20731 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b5ceea71b55a496bb2947def2bc4a9ae. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 05:14:19.244287 20999 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b5ceea71b55a496bb2947def2bc4a9ae. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 05:14:19.244230 20865 tablet_service.cc:1467] Processing CreateTablet for tablet b5ceea71b55a496bb2947def2bc4a9ae (DEFAULT_TABLE table=test-table [id=79973975394b4b0193859526d0a1bc37]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 05:14:19.244496 20865 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b5ceea71b55a496bb2947def2bc4a9ae. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 05:14:19.247953 21082 tablet_bootstrap.cc:492] T b5ceea71b55a496bb2947def2bc4a9ae P ed2350f926634e20b250a0c82a62251c: Bootstrap starting.
I20260621 05:14:19.248041 21083 tablet_bootstrap.cc:492] T b5ceea71b55a496bb2947def2bc4a9ae P 78a242bbee3d4eea8ce334817dfdfe5c: Bootstrap starting.
I20260621 05:14:19.248732 21082 tablet_bootstrap.cc:654] T b5ceea71b55a496bb2947def2bc4a9ae P ed2350f926634e20b250a0c82a62251c: Neither blocks nor log segments found. Creating new log.
I20260621 05:14:19.248901 21084 tablet_bootstrap.cc:492] T b5ceea71b55a496bb2947def2bc4a9ae P 745736f5f556409fa20a260fb647ab93: Bootstrap starting.
I20260621 05:14:19.249418 21082 log.cc:824] T b5ceea71b55a496bb2947def2bc4a9ae P ed2350f926634e20b250a0c82a62251c: Log is configured to *not* fsync() on all Append() calls
I20260621 05:14:19.249672 21084 tablet_bootstrap.cc:654] T b5ceea71b55a496bb2947def2bc4a9ae P 745736f5f556409fa20a260fb647ab93: Neither blocks nor log segments found. Creating new log.
I20260621 05:14:19.250310 21082 tablet_bootstrap.cc:492] T b5ceea71b55a496bb2947def2bc4a9ae P ed2350f926634e20b250a0c82a62251c: No bootstrap required, opened a new log
I20260621 05:14:19.250272 21083 tablet_bootstrap.cc:654] T b5ceea71b55a496bb2947def2bc4a9ae P 78a242bbee3d4eea8ce334817dfdfe5c: Neither blocks nor log segments found. Creating new log.
I20260621 05:14:19.250381 21082 ts_tablet_manager.cc:1397] T b5ceea71b55a496bb2947def2bc4a9ae P ed2350f926634e20b250a0c82a62251c: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260621 05:14:19.250967 21083 log.cc:824] T b5ceea71b55a496bb2947def2bc4a9ae P 78a242bbee3d4eea8ce334817dfdfe5c: Log is configured to *not* fsync() on all Append() calls
I20260621 05:14:19.251381 21084 log.cc:824] T b5ceea71b55a496bb2947def2bc4a9ae P 745736f5f556409fa20a260fb647ab93: Log is configured to *not* fsync() on all Append() calls
I20260621 05:14:19.251816 21083 tablet_bootstrap.cc:492] T b5ceea71b55a496bb2947def2bc4a9ae P 78a242bbee3d4eea8ce334817dfdfe5c: No bootstrap required, opened a new log
I20260621 05:14:19.251893 21083 ts_tablet_manager.cc:1397] T b5ceea71b55a496bb2947def2bc4a9ae P 78a242bbee3d4eea8ce334817dfdfe5c: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.001s
I20260621 05:14:19.252112 21084 tablet_bootstrap.cc:492] T b5ceea71b55a496bb2947def2bc4a9ae P 745736f5f556409fa20a260fb647ab93: No bootstrap required, opened a new log
I20260621 05:14:19.252190 21084 ts_tablet_manager.cc:1397] T b5ceea71b55a496bb2947def2bc4a9ae P 745736f5f556409fa20a260fb647ab93: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260621 05:14:19.252648 21082 raft_consensus.cc:348] T b5ceea71b55a496bb2947def2bc4a9ae P ed2350f926634e20b250a0c82a62251c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "745736f5f556409fa20a260fb647ab93" member_type: VOTER last_known_addr { host: "127.3.131.131" port: 42983 } } peers { permanent_uuid: "78a242bbee3d4eea8ce334817dfdfe5c" member_type: VOTER last_known_addr { host: "127.3.131.129" port: 40473 } } peers { permanent_uuid: "ed2350f926634e20b250a0c82a62251c" member_type: VOTER last_known_addr { host: "127.3.131.130" port: 42669 } }
I20260621 05:14:19.252838 21082 raft_consensus.cc:374] T b5ceea71b55a496bb2947def2bc4a9ae P ed2350f926634e20b250a0c82a62251c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 05:14:19.252875 21082 raft_consensus.cc:729] T b5ceea71b55a496bb2947def2bc4a9ae P ed2350f926634e20b250a0c82a62251c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ed2350f926634e20b250a0c82a62251c, State: Initialized, Role: FOLLOWER
I20260621 05:14:19.253023 21082 consensus_queue.cc:260] T b5ceea71b55a496bb2947def2bc4a9ae P ed2350f926634e20b250a0c82a62251c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "745736f5f556409fa20a260fb647ab93" member_type: VOTER last_known_addr { host: "127.3.131.131" port: 42983 } } peers { permanent_uuid: "78a242bbee3d4eea8ce334817dfdfe5c" member_type: VOTER last_known_addr { host: "127.3.131.129" port: 40473 } } peers { permanent_uuid: "ed2350f926634e20b250a0c82a62251c" member_type: VOTER last_known_addr { host: "127.3.131.130" port: 42669 } }
I20260621 05:14:19.253309 21082 ts_tablet_manager.cc:1428] T b5ceea71b55a496bb2947def2bc4a9ae P ed2350f926634e20b250a0c82a62251c: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260621 05:14:19.253474 20931 heartbeater.cc:503] Master 127.3.131.190:40619 was elected leader, sending a full tablet report...
I20260621 05:14:19.253854 21083 raft_consensus.cc:348] T b5ceea71b55a496bb2947def2bc4a9ae P 78a242bbee3d4eea8ce334817dfdfe5c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "745736f5f556409fa20a260fb647ab93" member_type: VOTER last_known_addr { host: "127.3.131.131" port: 42983 } } peers { permanent_uuid: "78a242bbee3d4eea8ce334817dfdfe5c" member_type: VOTER last_known_addr { host: "127.3.131.129" port: 40473 } } peers { permanent_uuid: "ed2350f926634e20b250a0c82a62251c" member_type: VOTER last_known_addr { host: "127.3.131.130" port: 42669 } }
I20260621 05:14:19.254000 21083 raft_consensus.cc:374] T b5ceea71b55a496bb2947def2bc4a9ae P 78a242bbee3d4eea8ce334817dfdfe5c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 05:14:19.254027 21083 raft_consensus.cc:729] T b5ceea71b55a496bb2947def2bc4a9ae P 78a242bbee3d4eea8ce334817dfdfe5c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 78a242bbee3d4eea8ce334817dfdfe5c, State: Initialized, Role: FOLLOWER
I20260621 05:14:19.254019 21084 raft_consensus.cc:348] T b5ceea71b55a496bb2947def2bc4a9ae P 745736f5f556409fa20a260fb647ab93 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "745736f5f556409fa20a260fb647ab93" member_type: VOTER last_known_addr { host: "127.3.131.131" port: 42983 } } peers { permanent_uuid: "78a242bbee3d4eea8ce334817dfdfe5c" member_type: VOTER last_known_addr { host: "127.3.131.129" port: 40473 } } peers { permanent_uuid: "ed2350f926634e20b250a0c82a62251c" member_type: VOTER last_known_addr { host: "127.3.131.130" port: 42669 } }
I20260621 05:14:19.254140 21083 consensus_queue.cc:260] T b5ceea71b55a496bb2947def2bc4a9ae P 78a242bbee3d4eea8ce334817dfdfe5c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "745736f5f556409fa20a260fb647ab93" member_type: VOTER last_known_addr { host: "127.3.131.131" port: 42983 } } peers { permanent_uuid: "78a242bbee3d4eea8ce334817dfdfe5c" member_type: VOTER last_known_addr { host: "127.3.131.129" port: 40473 } } peers { permanent_uuid: "ed2350f926634e20b250a0c82a62251c" member_type: VOTER last_known_addr { host: "127.3.131.130" port: 42669 } }
I20260621 05:14:19.254175 21084 raft_consensus.cc:374] T b5ceea71b55a496bb2947def2bc4a9ae P 745736f5f556409fa20a260fb647ab93 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 05:14:19.254215 21084 raft_consensus.cc:729] T b5ceea71b55a496bb2947def2bc4a9ae P 745736f5f556409fa20a260fb647ab93 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 745736f5f556409fa20a260fb647ab93, State: Initialized, Role: FOLLOWER
I20260621 05:14:19.254333 21084 consensus_queue.cc:260] T b5ceea71b55a496bb2947def2bc4a9ae P 745736f5f556409fa20a260fb647ab93 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "745736f5f556409fa20a260fb647ab93" member_type: VOTER last_known_addr { host: "127.3.131.131" port: 42983 } } peers { permanent_uuid: "78a242bbee3d4eea8ce334817dfdfe5c" member_type: VOTER last_known_addr { host: "127.3.131.129" port: 40473 } } peers { permanent_uuid: "ed2350f926634e20b250a0c82a62251c" member_type: VOTER last_known_addr { host: "127.3.131.130" port: 42669 } }
I20260621 05:14:19.254407 20797 heartbeater.cc:503] Master 127.3.131.190:40619 was elected leader, sending a full tablet report...
I20260621 05:14:19.254439 21083 ts_tablet_manager.cc:1428] T b5ceea71b55a496bb2947def2bc4a9ae P 78a242bbee3d4eea8ce334817dfdfe5c: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260621 05:14:19.254663 21084 ts_tablet_manager.cc:1428] T b5ceea71b55a496bb2947def2bc4a9ae P 745736f5f556409fa20a260fb647ab93: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260621 05:14:19.254823 21065 heartbeater.cc:503] Master 127.3.131.190:40619 was elected leader, sending a full tablet report...
I20260621 05:14:19.256007 21088 raft_consensus.cc:482] T b5ceea71b55a496bb2947def2bc4a9ae P ed2350f926634e20b250a0c82a62251c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260621 05:14:19.256086 21088 raft_consensus.cc:504] T b5ceea71b55a496bb2947def2bc4a9ae P ed2350f926634e20b250a0c82a62251c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "745736f5f556409fa20a260fb647ab93" member_type: VOTER last_known_addr { host: "127.3.131.131" port: 42983 } } peers { permanent_uuid: "78a242bbee3d4eea8ce334817dfdfe5c" member_type: VOTER last_known_addr { host: "127.3.131.129" port: 40473 } } peers { permanent_uuid: "ed2350f926634e20b250a0c82a62251c" member_type: VOTER last_known_addr { host: "127.3.131.130" port: 42669 } }
I20260621 05:14:19.256328 21088 leader_election.cc:288] T b5ceea71b55a496bb2947def2bc4a9ae P ed2350f926634e20b250a0c82a62251c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 745736f5f556409fa20a260fb647ab93 (127.3.131.131:42983), 78a242bbee3d4eea8ce334817dfdfe5c (127.3.131.129:40473)
I20260621 05:14:19.259032 21080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:14:19.256537 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.131.130:58433 (local address 127.3.131.131:42983)
0621 05:14:19.256580 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0621 05:14:19.256582 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:14:19.256726 (+ 144us) server_negotiation.cc:407] Connection header received
0621 05:14:19.256758 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:14:19.256759 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:14:19.256768 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:14:19.256827 (+ 59us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 05:14:19.256967 (+ 140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:14:19.257433 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:14:19.258350 (+ 917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:14:19.258702 (+ 352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:14:19.258758 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:14:19.258783 (+ 25us) server_negotiation.cc:1035] Waiting for connection context
0621 05:14:19.258925 (+ 142us) server_negotiation.cc:299] Negotiation successful
0621 05:14:19.258969 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260621 05:14:19.259063 21078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:14:19.256539 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.131.130:55343 (local address 127.3.131.129:40473)
0621 05:14:19.256614 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0621 05:14:19.256616 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:14:19.256646 (+ 30us) server_negotiation.cc:407] Connection header received
0621 05:14:19.256673 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:14:19.256674 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:14:19.256683 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:14:19.256737 (+ 54us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 05:14:19.256895 (+ 158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:14:19.257421 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:14:19.258311 (+ 890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:14:19.258712 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:14:19.258799 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:14:19.258807 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0621 05:14:19.258969 (+ 162us) server_negotiation.cc:299] Negotiation successful
0621 05:14:19.259006 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260621 05:14:19.259110 20933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:14:19.256481 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.131.129:40473 (local address 127.3.131.130:55343)
0621 05:14:19.256540 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0621 05:14:19.256552 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0621 05:14:19.256593 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:14:19.256750 (+ 157us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:14:19.256751 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:14:19.256752 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 05:14:19.256827 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:14:19.256828 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:14:19.257499 (+ 671us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:14:19.257500 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:14:19.258230 (+ 730us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:14:19.258232 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:14:19.258807 (+ 575us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:14:19.258808 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:14:19.258884 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:14:19.258887 (+ 3us) client_negotiation.cc:769] Sending connection context
0621 05:14:19.258957 (+ 70us) client_negotiation.cc:240] Negotiation successful
0621 05:14:19.258995 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":40}
I20260621 05:14:19.259061 21091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:14:19.256482 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.131.131:42983 (local address 127.3.131.130:58433)
0621 05:14:19.256646 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0621 05:14:19.256658 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0621 05:14:19.256718 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:14:19.256837 (+ 119us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:14:19.256838 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:14:19.256839 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 05:14:19.256915 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:14:19.256916 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:14:19.257526 (+ 610us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:14:19.257527 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:14:19.258252 (+ 725us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:14:19.258254 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:14:19.258780 (+ 526us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:14:19.258781 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:14:19.258854 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:14:19.258858 (+ 4us) client_negotiation.cc:769] Sending connection context
0621 05:14:19.258909 (+ 51us) client_negotiation.cc:240] Negotiation successful
0621 05:14:19.258954 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":146,"mutex_wait_us":20,"thread_start_us":95,"threads_started":1}
I20260621 05:14:19.259378 20751 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b5ceea71b55a496bb2947def2bc4a9ae" candidate_uuid: "ed2350f926634e20b250a0c82a62251c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78a242bbee3d4eea8ce334817dfdfe5c" is_pre_election: true
I20260621 05:14:19.259474 21019 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b5ceea71b55a496bb2947def2bc4a9ae" candidate_uuid: "ed2350f926634e20b250a0c82a62251c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "745736f5f556409fa20a260fb647ab93" is_pre_election: true
I20260621 05:14:19.259527 20751 raft_consensus.cc:2454] T b5ceea71b55a496bb2947def2bc4a9ae P 78a242bbee3d4eea8ce334817dfdfe5c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ed2350f926634e20b250a0c82a62251c in term 0.
I20260621 05:14:19.259594 21019 raft_consensus.cc:2454] T b5ceea71b55a496bb2947def2bc4a9ae P 745736f5f556409fa20a260fb647ab93 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ed2350f926634e20b250a0c82a62251c in term 0.
I20260621 05:14:19.259732 20818 leader_election.cc:302] T b5ceea71b55a496bb2947def2bc4a9ae P ed2350f926634e20b250a0c82a62251c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 78a242bbee3d4eea8ce334817dfdfe5c, ed2350f926634e20b250a0c82a62251c; no voters:
I20260621 05:14:19.259866 21088 raft_consensus.cc:2781] T b5ceea71b55a496bb2947def2bc4a9ae P ed2350f926634e20b250a0c82a62251c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260621 05:14:19.259930 21088 raft_consensus.cc:482] T b5ceea71b55a496bb2947def2bc4a9ae P ed2350f926634e20b250a0c82a62251c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260621 05:14:19.259968 21088 raft_consensus.cc:3037] T b5ceea71b55a496bb2947def2bc4a9ae P ed2350f926634e20b250a0c82a62251c [term 0 FOLLOWER]: Advancing to term 1
I20260621 05:14:19.260650 21088 raft_consensus.cc:504] T b5ceea71b55a496bb2947def2bc4a9ae P ed2350f926634e20b250a0c82a62251c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "745736f5f556409fa20a260fb647ab93" member_type: VOTER last_known_addr { host: "127.3.131.131" port: 42983 } } peers { permanent_uuid: "78a242bbee3d4eea8ce334817dfdfe5c" member_type: VOTER last_known_addr { host: "127.3.131.129" port: 40473 } } peers { permanent_uuid: "ed2350f926634e20b250a0c82a62251c" member_type: VOTER last_known_addr { host: "127.3.131.130" port: 42669 } }
I20260621 05:14:19.260788 21088 leader_election.cc:288] T b5ceea71b55a496bb2947def2bc4a9ae P ed2350f926634e20b250a0c82a62251c [CANDIDATE]: Term 1 election: Requested vote from peers 745736f5f556409fa20a260fb647ab93 (127.3.131.131:42983), 78a242bbee3d4eea8ce334817dfdfe5c (127.3.131.129:40473)
I20260621 05:14:19.260921 21019 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b5ceea71b55a496bb2947def2bc4a9ae" candidate_uuid: "ed2350f926634e20b250a0c82a62251c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "745736f5f556409fa20a260fb647ab93"
I20260621 05:14:19.261004 21019 raft_consensus.cc:3037] T b5ceea71b55a496bb2947def2bc4a9ae P 745736f5f556409fa20a260fb647ab93 [term 0 FOLLOWER]: Advancing to term 1
I20260621 05:14:19.260991 20751 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b5ceea71b55a496bb2947def2bc4a9ae" candidate_uuid: "ed2350f926634e20b250a0c82a62251c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78a242bbee3d4eea8ce334817dfdfe5c"
I20260621 05:14:19.261063 20751 raft_consensus.cc:3037] T b5ceea71b55a496bb2947def2bc4a9ae P 78a242bbee3d4eea8ce334817dfdfe5c [term 0 FOLLOWER]: Advancing to term 1
I20260621 05:14:19.261701 21019 raft_consensus.cc:2454] T b5ceea71b55a496bb2947def2bc4a9ae P 745736f5f556409fa20a260fb647ab93 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ed2350f926634e20b250a0c82a62251c in term 1.
I20260621 05:14:19.261757 20751 raft_consensus.cc:2454] T b5ceea71b55a496bb2947def2bc4a9ae P 78a242bbee3d4eea8ce334817dfdfe5c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ed2350f926634e20b250a0c82a62251c in term 1.
I20260621 05:14:19.261900 20819 leader_election.cc:302] T b5ceea71b55a496bb2947def2bc4a9ae P ed2350f926634e20b250a0c82a62251c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 745736f5f556409fa20a260fb647ab93, ed2350f926634e20b250a0c82a62251c; no voters:
I20260621 05:14:19.261997 21088 raft_consensus.cc:2781] T b5ceea71b55a496bb2947def2bc4a9ae P ed2350f926634e20b250a0c82a62251c [term 1 FOLLOWER]: Leader election won for term 1
I20260621 05:14:19.262120 21088 raft_consensus.cc:686] T b5ceea71b55a496bb2947def2bc4a9ae P ed2350f926634e20b250a0c82a62251c [term 1 LEADER]: Becoming Leader. State: Replica: ed2350f926634e20b250a0c82a62251c, State: Running, Role: LEADER
I20260621 05:14:19.262224 21088 consensus_queue.cc:237] T b5ceea71b55a496bb2947def2bc4a9ae P ed2350f926634e20b250a0c82a62251c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "745736f5f556409fa20a260fb647ab93" member_type: VOTER last_known_addr { host: "127.3.131.131" port: 42983 } } peers { permanent_uuid: "78a242bbee3d4eea8ce334817dfdfe5c" member_type: VOTER last_known_addr { host: "127.3.131.129" port: 40473 } } peers { permanent_uuid: "ed2350f926634e20b250a0c82a62251c" member_type: VOTER last_known_addr { host: "127.3.131.130" port: 42669 } }
I20260621 05:14:19.263041 20604 catalog_manager.cc:5495] T b5ceea71b55a496bb2947def2bc4a9ae P ed2350f926634e20b250a0c82a62251c reported cstate change: term changed from 0 to 1, leader changed from <none> to ed2350f926634e20b250a0c82a62251c (127.3.131.130). New cstate: current_term: 1 leader_uuid: "ed2350f926634e20b250a0c82a62251c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "745736f5f556409fa20a260fb647ab93" member_type: VOTER last_known_addr { host: "127.3.131.131" port: 42983 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "78a242bbee3d4eea8ce334817dfdfe5c" member_type: VOTER last_known_addr { host: "127.3.131.129" port: 40473 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ed2350f926634e20b250a0c82a62251c" member_type: VOTER last_known_addr { host: "127.3.131.130" port: 42669 } health_report { overall_health: HEALTHY } } }
I20260621 05:14:19.272102 21081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:14:19.269257 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58422 (local address 127.3.131.130:42669)
0621 05:14:19.269355 (+ 98us) server_negotiation.cc:206] Beginning negotiation
0621 05:14:19.269357 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:14:19.269397 (+ 40us) server_negotiation.cc:407] Connection header received
0621 05:14:19.269495 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:14:19.269501 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:14:19.269518 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:14:19.269594 (+ 76us) server_negotiation.cc:226] Negotiated authn=TOKEN
0621 05:14:19.269775 (+ 181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:14:19.270270 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:14:19.271042 (+ 772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:14:19.271321 (+ 279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:14:19.271406 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:14:19.271578 (+ 172us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0621 05:14:19.271788 (+ 210us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0621 05:14:19.271924 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0621 05:14:19.271943 (+ 19us) server_negotiation.cc:299] Negotiation successful
0621 05:14:19.271968 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260621 05:14:19.274178 20751 raft_consensus.cc:1261] T b5ceea71b55a496bb2947def2bc4a9ae P 78a242bbee3d4eea8ce334817dfdfe5c [term 1 FOLLOWER]: Refusing update from remote peer ed2350f926634e20b250a0c82a62251c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 05:14:19.274178 21019 raft_consensus.cc:1261] T b5ceea71b55a496bb2947def2bc4a9ae P 745736f5f556409fa20a260fb647ab93 [term 1 FOLLOWER]: Refusing update from remote peer ed2350f926634e20b250a0c82a62251c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 05:14:19.274560 21088 consensus_queue.cc:1035] T b5ceea71b55a496bb2947def2bc4a9ae P ed2350f926634e20b250a0c82a62251c [LEADER]: Connected to new peer: Peer: permanent_uuid: "78a242bbee3d4eea8ce334817dfdfe5c" member_type: VOTER last_known_addr { host: "127.3.131.129" port: 40473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 05:14:19.274757 21092 consensus_queue.cc:1035] T b5ceea71b55a496bb2947def2bc4a9ae P ed2350f926634e20b250a0c82a62251c [LEADER]: Connected to new peer: Peer: permanent_uuid: "745736f5f556409fa20a260fb647ab93" member_type: VOTER last_known_addr { host: "127.3.131.131" port: 42983 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 05:14:19.278333 21095 mvcc.cc:205] Tried to move back new op lower bound from 7299149247583522816 to 7299149247538683904. Current Snapshot: MvccSnapshot[applied={T|T < 7299149247583522816}]
I20260621 05:14:19.278795 21097 mvcc.cc:205] Tried to move back new op lower bound from 7299149247583522816 to 7299149247538683904. Current Snapshot: MvccSnapshot[applied={T|T < 7299149247583522816}]
I20260621 05:14:19.281090 20606 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45962:
table { table_name: "test-table" } modify_external_catalogs: true
I20260621 05:14:19.281227 20606 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45962:
table { table_name: "test-table" } modify_external_catalogs: true
I20260621 05:14:19.282306 20606 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ab7cef70ff3946869f3eaf8b05d0147d: Sending DeleteTablet for 3 replicas of tablet b5ceea71b55a496bb2947def2bc4a9ae
I20260621 05:14:19.282711 20865 tablet_service.cc:1514] Processing DeleteTablet for tablet b5ceea71b55a496bb2947def2bc4a9ae with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 05:14:19 UTC) from {username='kudu', principal='kudu/127.3.131.190@KRBTEST.COM'} at 127.0.0.1:58416
I20260621 05:14:19.282691 20999 tablet_service.cc:1514] Processing DeleteTablet for tablet b5ceea71b55a496bb2947def2bc4a9ae with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 05:14:19 UTC) from {username='kudu', principal='kudu/127.3.131.190@KRBTEST.COM'} at 127.0.0.1:49442
I20260621 05:14:19.282730 20731 tablet_service.cc:1514] Processing DeleteTablet for tablet b5ceea71b55a496bb2947def2bc4a9ae with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 05:14:19 UTC) from {username='kudu', principal='kudu/127.3.131.190@KRBTEST.COM'} at 127.0.0.1:35076
I20260621 05:14:19.282959 21106 tablet_replica.cc:330] T b5ceea71b55a496bb2947def2bc4a9ae P 745736f5f556409fa20a260fb647ab93: stopping tablet replica
I20260621 05:14:19.283070 21106 raft_consensus.cc:2229] T b5ceea71b55a496bb2947def2bc4a9ae P 745736f5f556409fa20a260fb647ab93 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260621 05:14:19.283226 21106 raft_consensus.cc:2258] T b5ceea71b55a496bb2947def2bc4a9ae P 745736f5f556409fa20a260fb647ab93 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 05:14:19.283274 21107 tablet_replica.cc:330] T b5ceea71b55a496bb2947def2bc4a9ae P 78a242bbee3d4eea8ce334817dfdfe5c: stopping tablet replica
I20260621 05:14:19.283337 3598 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWZOx2_/build/release/bin/kudu with pid 20668
I20260621 05:14:19.283418 21107 raft_consensus.cc:2229] T b5ceea71b55a496bb2947def2bc4a9ae P 78a242bbee3d4eea8ce334817dfdfe5c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260621 05:14:19.283537 21107 raft_consensus.cc:2258] T b5ceea71b55a496bb2947def2bc4a9ae P 78a242bbee3d4eea8ce334817dfdfe5c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 05:14:19.284138 21106 ts_tablet_manager.cc:1905] T b5ceea71b55a496bb2947def2bc4a9ae P 745736f5f556409fa20a260fb647ab93: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 05:14:19.285919 21106 ts_tablet_manager.cc:1918] T b5ceea71b55a496bb2947def2bc4a9ae P 745736f5f556409fa20a260fb647ab93: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260621 05:14:19.285987 21106 log.cc:1196] T b5ceea71b55a496bb2947def2bc4a9ae P 745736f5f556409fa20a260fb647ab93: Deleting WAL directory at /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782018616648578-3598-0/minicluster-data/ts-2/wal/wals/b5ceea71b55a496bb2947def2bc4a9ae
I20260621 05:14:19.286281 21106 ts_tablet_manager.cc:1939] T b5ceea71b55a496bb2947def2bc4a9ae P 745736f5f556409fa20a260fb647ab93: Deleting consensus metadata
I20260621 05:14:19.286731 20591 catalog_manager.cc:4841] TS 745736f5f556409fa20a260fb647ab93 (127.3.131.131:42983): tablet b5ceea71b55a496bb2947def2bc4a9ae (table test-table [id=79973975394b4b0193859526d0a1bc37]) successfully deleted
W20260621 05:14:19.289942 20590 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.3.131.129:40473 (error 108)
I20260621 05:14:19.290267 21105 tablet_replica.cc:330] T b5ceea71b55a496bb2947def2bc4a9ae P ed2350f926634e20b250a0c82a62251c: stopping tablet replica
I20260621 05:14:19.290475 21105 raft_consensus.cc:2229] T b5ceea71b55a496bb2947def2bc4a9ae P ed2350f926634e20b250a0c82a62251c [term 1 LEADER]: Raft consensus shutting down.
I20260621 05:14:19.290585 21077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:14:19.290433 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.131.129:40473 (local address 127.0.0.1:35090)
0621 05:14:19.290484 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0621 05:14:19.290530 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.131.129:40473: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":24}
I20260621 05:14:19.290603 3598 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWZOx2_/build/release/bin/kudu with pid 20802
I20260621 05:14:19.290673 21105 raft_consensus.cc:2258] T b5ceea71b55a496bb2947def2bc4a9ae P ed2350f926634e20b250a0c82a62251c [term 1 FOLLOWER]: Raft consensus is shut down!
W20260621 05:14:19.290722 20590 catalog_manager.cc:4584] TS 78a242bbee3d4eea8ce334817dfdfe5c (127.3.131.129:40473): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet b5ceea71b55a496bb2947def2bc4a9ae: Network error: Client connection negotiation failed: client connection to 127.3.131.129:40473: connect: Connection refused (error 111)
I20260621 05:14:19.296763 21077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:14:19.296571 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.131.130:42669 (local address 127.0.0.1:58430)
0621 05:14:19.296626 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0621 05:14:19.296703 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.131.130:42669: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":24}
I20260621 05:14:19.296798 3598 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWZOx2_/build/release/bin/kudu with pid 20936
I20260621 05:14:19.303586 3598 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWZOx2_/build/release/bin/kudu with pid 20574
2026-06-21T05:14:19Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (1991 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-06-21T05:14:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-21T05:14:19Z Disabled control of system clock
I20260621 05:14:19.333806 3598 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWZOx2_/build/release/bin/kudu
/tmp/dist-test-taskWZOx2_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782018616648578-3598-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782018616648578-3598-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782018616648578-3598-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782018616648578-3598-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.131.190:38169
--webserver_interface=127.3.131.190
--webserver_port=0
--builtin_ntp_servers=127.3.131.148:42409
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.131.190:38169
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260621 05:14:19.420327 21112 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 05:14:19.420625 21112 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 05:14:19.420665 21112 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 05:14:19.421908 21112 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 05:14:19.421977 21112 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 05:14:19.421998 21112 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 05:14:19.422017 21112 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 05:14:19.422046 21112 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 05:14:19.423591 21112 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.131.148:42409
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782018616648578-3598-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782018616648578-3598-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.131.190:38169
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.131.190:38169
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782018616648578-3598-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.131.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21112
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782018616648578-3598-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 21 Jun 2026 04:16:16 UTC on ccycloud.quasar-xycaha.root.comops.site
build id 4457881
I20260621 05:14:19.423898 21112 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 05:14:19.424219 21112 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 05:14:19.429168 21118 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:14:19.429177 21117 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:14:19.429276 21120 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:14:19.429394 21112 server_base.cc:1034] running on GCE node
I20260621 05:14:19.429756 21112 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 05:14:19.430106 21112 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 05:14:19.431262 21112 hybrid_clock.cc:648] HybridClock initialized: now 1782018859431239 us; error 38 us; skew 500 ppm
I20260621 05:14:19.432606 21112 webserver.cc:460] Webserver started at http://127.3.131.190:43985/ using document root <none> and password file <none>
I20260621 05:14:19.432879 21112 fs_manager.cc:359] Metadata directory not provided
I20260621 05:14:19.432951 21112 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 05:14:19.433099 21112 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 05:14:19.434259 21112 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782018616648578-3598-0/minicluster-data/master-0/data/instance:
uuid: "07cafeb58da04f02806c39abc67c24f1"
format_stamp: "Formatted at 2026-06-21 05:14:19 on dist-test-slave-vrq1"
server_key: "d6e1ac2e26f035d729c35b1db6fb2ea0"
server_key_iv: "82af9316bb931dbe1ec75a1841f6bd2f"
server_key_version: "encryptionkey@0"
I20260621 05:14:19.434620 21112 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782018616648578-3598-0/minicluster-data/master-0/wal/instance:
uuid: "07cafeb58da04f02806c39abc67c24f1"
format_stamp: "Formatted at 2026-06-21 05:14:19 on dist-test-slave-vrq1"
server_key: "d6e1ac2e26f035d729c35b1db6fb2ea0"
server_key_iv: "82af9316bb931dbe1ec75a1841f6bd2f"
server_key_version: "encryptionkey@0"
I20260621 05:14:19.436112 21112 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260621 05:14:19.437009 21126 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 05:14:19.437254 21112 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260621 05:14:19.437372 21112 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782018616648578-3598-0/minicluster-data/master-0/data,/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782018616648578-3598-0/minicluster-data/master-0/wal
uuid: "07cafeb58da04f02806c39abc67c24f1"
format_stamp: "Formatted at 2026-06-21 05:14:19 on dist-test-slave-vrq1"
server_key: "d6e1ac2e26f035d729c35b1db6fb2ea0"
server_key_iv: "82af9316bb931dbe1ec75a1841f6bd2f"
server_key_version: "encryptionkey@0"
I20260621 05:14:19.437454 21112 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782018616648578-3598-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782018616648578-3598-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782018616648578-3598-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 05:14:19.460681 21112 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 05:14:19.461052 21112 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 05:14:19.461217 21112 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 05:14:19.465438 21112 rpc_server.cc:307] RPC server started. Bound to: 127.3.131.190:38169
I20260621 05:14:19.465539 21178 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.131.190:38169 every 8 connection(s)
I20260621 05:14:19.465880 21112 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782018616648578-3598-0/minicluster-data/master-0/data/info.pb
I20260621 05:14:19.467008 21179 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 05:14:19.468779 3598 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWZOx2_/build/release/bin/kudu as pid 21112
I20260621 05:14:19.468905 3598 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782018616648578-3598-0/minicluster-data/master-0/wal/instance
I20260621 05:14:19.469085 3598 external_mini_cluster.cc:1428] Setting key fccb86040cda1ffd03e971379cd1048a
I20260621 05:14:19.469726 21179 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 07cafeb58da04f02806c39abc67c24f1: Bootstrap starting.
I20260621 05:14:19.470338 21179 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 07cafeb58da04f02806c39abc67c24f1: Neither blocks nor log segments found. Creating new log.
I20260621 05:14:19.470654 21179 log.cc:824] T 00000000000000000000000000000000 P 07cafeb58da04f02806c39abc67c24f1: Log is configured to *not* fsync() on all Append() calls
I20260621 05:14:19.471412 21179 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 07cafeb58da04f02806c39abc67c24f1: No bootstrap required, opened a new log
I20260621 05:14:19.473548 21179 raft_consensus.cc:348] T 00000000000000000000000000000000 P 07cafeb58da04f02806c39abc67c24f1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "07cafeb58da04f02806c39abc67c24f1" member_type: VOTER last_known_addr { host: "127.3.131.190" port: 38169 } }
I20260621 05:14:19.473766 21179 raft_consensus.cc:374] T 00000000000000000000000000000000 P 07cafeb58da04f02806c39abc67c24f1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 05:14:19.473802 21179 raft_consensus.cc:729] T 00000000000000000000000000000000 P 07cafeb58da04f02806c39abc67c24f1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 07cafeb58da04f02806c39abc67c24f1, State: Initialized, Role: FOLLOWER
I20260621 05:14:19.473929 21179 consensus_queue.cc:260] T 00000000000000000000000000000000 P 07cafeb58da04f02806c39abc67c24f1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "07cafeb58da04f02806c39abc67c24f1" member_type: VOTER last_known_addr { host: "127.3.131.190" port: 38169 } }
I20260621 05:14:19.474043 21179 raft_consensus.cc:388] T 00000000000000000000000000000000 P 07cafeb58da04f02806c39abc67c24f1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260621 05:14:19.474085 21179 raft_consensus.cc:482] T 00000000000000000000000000000000 P 07cafeb58da04f02806c39abc67c24f1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260621 05:14:19.474133 21179 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 07cafeb58da04f02806c39abc67c24f1 [term 0 FOLLOWER]: Advancing to term 1
I20260621 05:14:19.474762 21179 raft_consensus.cc:504] T 00000000000000000000000000000000 P 07cafeb58da04f02806c39abc67c24f1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "07cafeb58da04f02806c39abc67c24f1" member_type: VOTER last_known_addr { host: "127.3.131.190" port: 38169 } }
I20260621 05:14:19.474900 21179 leader_election.cc:302] T 00000000000000000000000000000000 P 07cafeb58da04f02806c39abc67c24f1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 07cafeb58da04f02806c39abc67c24f1; no voters:
I20260621 05:14:19.475147 21179 leader_election.cc:288] T 00000000000000000000000000000000 P 07cafeb58da04f02806c39abc67c24f1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260621 05:14:19.475246 21184 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 07cafeb58da04f02806c39abc67c24f1 [term 1 FOLLOWER]: Leader election won for term 1
I20260621 05:14:19.475411 21184 raft_consensus.cc:686] T 00000000000000000000000000000000 P 07cafeb58da04f02806c39abc67c24f1 [term 1 LEADER]: Becoming Leader. State: Replica: 07cafeb58da04f02806c39abc67c24f1, State: Running, Role: LEADER
I20260621 05:14:19.475571 21184 consensus_queue.cc:237] T 00000000000000000000000000000000 P 07cafeb58da04f02806c39abc67c24f1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "07cafeb58da04f02806c39abc67c24f1" member_type: VOTER last_known_addr { host: "127.3.131.190" port: 38169 } }
I20260621 05:14:19.475571 21179 sys_catalog.cc:564] T 00000000000000000000000000000000 P 07cafeb58da04f02806c39abc67c24f1 [sys.catalog]: configured and running, proceeding with master startup.
I20260621 05:14:19.475960 21186 sys_catalog.cc:455] T 00000000000000000000000000000000 P 07cafeb58da04f02806c39abc67c24f1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 07cafeb58da04f02806c39abc67c24f1. Latest consensus state: current_term: 1 leader_uuid: "07cafeb58da04f02806c39abc67c24f1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "07cafeb58da04f02806c39abc67c24f1" member_type: VOTER last_known_addr { host: "127.3.131.190" port: 38169 } } }
I20260621 05:14:19.476069 21186 sys_catalog.cc:458] T 00000000000000000000000000000000 P 07cafeb58da04f02806c39abc67c24f1 [sys.catalog]: This master's current role is: LEADER
I20260621 05:14:19.476667 21182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:14:19.469692 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33092 (local address 127.3.131.190:38169)
0621 05:14:19.469906 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0621 05:14:19.469921 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0621 05:14:19.471034 (+ 1113us) server_negotiation.cc:407] Connection header received
0621 05:14:19.471622 (+ 588us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:14:19.471631 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:14:19.471706 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:14:19.471846 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:14:19.472136 (+ 290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:14:19.473096 (+ 960us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:14:19.474267 (+ 1171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:14:19.474756 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:14:19.474875 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:14:19.475716 (+ 841us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:14:19.475720 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:14:19.475720 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 05:14:19.475950 (+ 230us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:14:19.475993 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 05:14:19.476008 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:14:19.476098 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0621 05:14:19.476327 (+ 229us) server_negotiation.cc:299] Negotiation successful
0621 05:14:19.476431 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":128,"thread_start_us":98,"threads_started":1}
I20260621 05:14:19.476848 21185 sys_catalog.cc:455] T 00000000000000000000000000000000 P 07cafeb58da04f02806c39abc67c24f1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "07cafeb58da04f02806c39abc67c24f1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "07cafeb58da04f02806c39abc67c24f1" member_type: VOTER last_known_addr { host: "127.3.131.190" port: 38169 } } }
I20260621 05:14:19.476950 21185 sys_catalog.cc:458] T 00000000000000000000000000000000 P 07cafeb58da04f02806c39abc67c24f1 [sys.catalog]: This master's current role is: LEADER
I20260621 05:14:19.477283 21188 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260621 05:14:19.477802 21188 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260621 05:14:19.479295 21188 catalog_manager.cc:1347] Generated new cluster ID: 639cd7e6b93348a98f8bf7f41355c61b
I20260621 05:14:19.479338 21188 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260621 05:14:19.488056 21188 catalog_manager.cc:1370] Generated new certificate authority record
I20260621 05:14:19.488701 21188 catalog_manager.cc:1504] Loading token signing keys...
I20260621 05:14:19.498956 21188 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 07cafeb58da04f02806c39abc67c24f1: Generated new TSK 0
I20260621 05:14:19.499344 21188 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260621 05:14:19.505944 3598 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWZOx2_/build/release/bin/kudu
/tmp/dist-test-taskWZOx2_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782018616648578-3598-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782018616648578-3598-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782018616648578-3598-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782018616648578-3598-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.131.129:0
--local_ip_for_outbound_sockets=127.3.131.129
--webserver_interface=127.3.131.129
--webserver_port=0
--tserver_master_addrs=127.3.131.190:38169
--builtin_ntp_servers=127.3.131.148:42409
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260621 05:14:19.593485 21203 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 05:14:19.593721 21203 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 05:14:19.593745 21203 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 05:14:19.595068 21203 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 05:14:19.595124 21203 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 05:14:19.595152 21203 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.131.129
I20260621 05:14:19.596654 21203 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.131.148:42409
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782018616648578-3598-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782018616648578-3598-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.131.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782018616648578-3598-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.131.129
--webserver_port=0
--tserver_master_addrs=127.3.131.190:38169
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21203
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.131.129
--log_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782018616648578-3598-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 21 Jun 2026 04:16:16 UTC on ccycloud.quasar-xycaha.root.comops.site
build id 4457881
I20260621 05:14:19.596944 21203 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 05:14:19.597275 21203 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 05:14:19.603294 21211 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:14:19.603298 21208 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:14:19.603294 21209 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:14:19.603451 21203 server_base.cc:1034] running on GCE node
I20260621 05:14:19.603896 21203 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 05:14:19.604182 21203 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 05:14:19.605392 21203 hybrid_clock.cc:648] HybridClock initialized: now 1782018859605373 us; error 29 us; skew 500 ppm
I20260621 05:14:19.606890 21203 webserver.cc:460] Webserver started at http://127.3.131.129:46189/ using document root <none> and password file <none>
I20260621 05:14:19.607234 21203 fs_manager.cc:359] Metadata directory not provided
I20260621 05:14:19.607309 21203 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 05:14:19.607468 21203 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 05:14:19.608634 21203 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782018616648578-3598-0/minicluster-data/ts-0/data/instance:
uuid: "04f53e20f5cb4d6c9250fe7c4945913d"
format_stamp: "Formatted at 2026-06-21 05:14:19 on dist-test-slave-vrq1"
server_key: "cc86529465a146eabd148d5835be80bc"
server_key_iv: "c4750c75e556507de5d537691c2e5c6f"
server_key_version: "encryptionkey@0"
I20260621 05:14:19.609006 21203 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782018616648578-3598-0/minicluster-data/ts-0/wal/instance:
uuid: "04f53e20f5cb4d6c9250fe7c4945913d"
format_stamp: "Formatted at 2026-06-21 05:14:19 on dist-test-slave-vrq1"
server_key: "cc86529465a146eabd148d5835be80bc"
server_key_iv: "c4750c75e556507de5d537691c2e5c6f"
server_key_version: "encryptionkey@0"
I20260621 05:14:19.610411 21203 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260621 05:14:19.611454 21217 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 05:14:19.611692 21203 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260621 05:14:19.611807 21203 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782018616648578-3598-0/minicluster-data/ts-0/data,/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782018616648578-3598-0/minicluster-data/ts-0/wal
uuid: "04f53e20f5cb4d6c9250fe7c4945913d"
format_stamp: "Formatted at 2026-06-21 05:14:19 on dist-test-slave-vrq1"
server_key: "cc86529465a146eabd148d5835be80bc"
server_key_iv: "c4750c75e556507de5d537691c2e5c6f"
server_key_version: "encryptionkey@0"
I20260621 05:14:19.611896 21203 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782018616648578-3598-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782018616648578-3598-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782018616648578-3598-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 05:14:19.622432 21203 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 05:14:19.622771 21203 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 05:14:19.622925 21203 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 05:14:19.623230 21203 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 05:14:19.623589 21203 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 05:14:19.623642 21203 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 05:14:19.623687 21203 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 05:14:19.623713 21203 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 05:14:19.630188 21203 rpc_server.cc:307] RPC server started. Bound to: 127.3.131.129:38477
I20260621 05:14:19.630227 21330 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.131.129:38477 every 8 connection(s)
I20260621 05:14:19.630576 21203 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782018616648578-3598-0/minicluster-data/ts-0/data/info.pb
I20260621 05:14:19.631381 3598 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWZOx2_/build/release/bin/kudu as pid 21203
I20260621 05:14:19.631479 3598 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782018616648578-3598-0/minicluster-data/ts-0/wal/instance
I20260621 05:14:19.631695 3598 external_mini_cluster.cc:1428] Setting key e6ac78be4f8b6cc0973ea7721f94aa96
I20260621 05:14:19.632664 3598 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWZOx2_/build/release/bin/kudu
/tmp/dist-test-taskWZOx2_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782018616648578-3598-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782018616648578-3598-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782018616648578-3598-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782018616648578-3598-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.131.130:0
--local_ip_for_outbound_sockets=127.3.131.130
--webserver_interface=127.3.131.130
--webserver_port=0
--tserver_master_addrs=127.3.131.190:38169
--builtin_ntp_servers=127.3.131.148:42409
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260621 05:14:19.636534 21182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:14:19.631588 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.131.129:58273 (local address 127.3.131.190:38169)
0621 05:14:19.631659 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0621 05:14:19.631661 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:14:19.631722 (+ 61us) server_negotiation.cc:407] Connection header received
0621 05:14:19.632207 (+ 485us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:14:19.632208 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:14:19.632220 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:14:19.632287 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:14:19.633057 (+ 770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:14:19.633831 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:14:19.634930 (+ 1099us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:14:19.635152 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:14:19.635277 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:14:19.636255 (+ 978us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:14:19.636257 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:14:19.636257 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 05:14:19.636263 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:14:19.636280 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 05:14:19.636284 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:14:19.636331 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0621 05:14:19.636419 (+ 88us) server_negotiation.cc:299] Negotiation successful
0621 05:14:19.636450 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260621 05:14:19.636763 21333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:14:19.631335 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.131.190:38169 (local address 127.3.131.129:58273)
0621 05:14:19.631520 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0621 05:14:19.631563 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0621 05:14:19.632113 (+ 550us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:14:19.632367 (+ 254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:14:19.632372 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:14:19.632383 (+ 11us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:14:19.632964 (+ 581us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:14:19.632969 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:14:19.633945 (+ 976us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:14:19.633947 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:14:19.634817 (+ 870us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:14:19.634820 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:14:19.635292 (+ 472us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:14:19.635293 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:14:19.635437 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:14:19.636087 (+ 650us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0621 05:14:19.636087 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:14:19.636120 (+ 33us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0621 05:14:19.636146 (+ 26us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 05:14:19.636331 (+ 185us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 05:14:19.636337 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 05:14:19.636340 (+ 3us) client_negotiation.cc:769] Sending connection context
0621 05:14:19.636424 (+ 84us) client_negotiation.cc:240] Negotiation successful
0621 05:14:19.636537 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":89,"threads_started":1}
I20260621 05:14:19.637331 21331 heartbeater.cc:347] Connected to a master server at 127.3.131.190:38169
I20260621 05:14:19.637445 21331 heartbeater.cc:464] Registering TS with master...
I20260621 05:14:19.637630 21331 heartbeater.cc:511] Master 127.3.131.190:38169 requested a full tablet report, sending...
I20260621 05:14:19.638208 21143 ts_manager.cc:194] Registered new tserver with Master: 04f53e20f5cb4d6c9250fe7c4945913d (127.3.131.129:38477)
I20260621 05:14:19.639282 21143 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.3.131.129:58273
W20260621 05:14:19.717072 21334 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 05:14:19.717293 21334 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 05:14:19.717317 21334 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 05:14:19.718544 21334 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 05:14:19.718590 21334 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 05:14:19.718616 21334 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.131.130
I20260621 05:14:19.720132 21334 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.131.148:42409
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782018616648578-3598-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782018616648578-3598-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.131.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782018616648578-3598-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.131.130
--webserver_port=0
--tserver_master_addrs=127.3.131.190:38169
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21334
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.131.130
--log_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782018616648578-3598-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 21 Jun 2026 04:16:16 UTC on ccycloud.quasar-xycaha.root.comops.site
build id 4457881
I20260621 05:14:19.720340 21334 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 05:14:19.720597 21334 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 05:14:19.725903 21342 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:14:19.725968 21340 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:14:19.726066 21334 server_base.cc:1034] running on GCE node
W20260621 05:14:19.725916 21339 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:14:19.726454 21334 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 05:14:19.726718 21334 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 05:14:19.727887 21334 hybrid_clock.cc:648] HybridClock initialized: now 1782018859727869 us; error 36 us; skew 500 ppm
I20260621 05:14:19.729244 21334 webserver.cc:460] Webserver started at http://127.3.131.130:35567/ using document root <none> and password file <none>
I20260621 05:14:19.729506 21334 fs_manager.cc:359] Metadata directory not provided
I20260621 05:14:19.729574 21334 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 05:14:19.729714 21334 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 05:14:19.730808 21334 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782018616648578-3598-0/minicluster-data/ts-1/data/instance:
uuid: "7faa5bd5726e4ae8a9ed344a6106f01f"
format_stamp: "Formatted at 2026-06-21 05:14:19 on dist-test-slave-vrq1"
server_key: "5f92c24839ca0218cfc1a50ad839cb9f"
server_key_iv: "e15463b6f54759510dc374bbfcc1111e"
server_key_version: "encryptionkey@0"
I20260621 05:14:19.731235 21334 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782018616648578-3598-0/minicluster-data/ts-1/wal/instance:
uuid: "7faa5bd5726e4ae8a9ed344a6106f01f"
format_stamp: "Formatted at 2026-06-21 05:14:19 on dist-test-slave-vrq1"
server_key: "5f92c24839ca0218cfc1a50ad839cb9f"
server_key_iv: "e15463b6f54759510dc374bbfcc1111e"
server_key_version: "encryptionkey@0"
I20260621 05:14:19.732640 21334 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260621 05:14:19.733412 21348 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 05:14:19.733603 21334 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260621 05:14:19.733700 21334 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782018616648578-3598-0/minicluster-data/ts-1/data,/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782018616648578-3598-0/minicluster-data/ts-1/wal
uuid: "7faa5bd5726e4ae8a9ed344a6106f01f"
format_stamp: "Formatted at 2026-06-21 05:14:19 on dist-test-slave-vrq1"
server_key: "5f92c24839ca0218cfc1a50ad839cb9f"
server_key_iv: "e15463b6f54759510dc374bbfcc1111e"
server_key_version: "encryptionkey@0"
I20260621 05:14:19.733794 21334 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782018616648578-3598-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782018616648578-3598-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782018616648578-3598-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 05:14:19.746153 21334 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 05:14:19.746466 21334 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 05:14:19.746609 21334 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 05:14:19.746858 21334 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 05:14:19.747251 21334 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 05:14:19.747303 21334 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 05:14:19.747345 21334 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 05:14:19.747372 21334 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 05:14:19.753191 21334 rpc_server.cc:307] RPC server started. Bound to: 127.3.131.130:39315
I20260621 05:14:19.753232 21461 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.131.130:39315 every 8 connection(s)
I20260621 05:14:19.753599 21334 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782018616648578-3598-0/minicluster-data/ts-1/data/info.pb
I20260621 05:14:19.758309 3598 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWZOx2_/build/release/bin/kudu as pid 21334
I20260621 05:14:19.758409 3598 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782018616648578-3598-0/minicluster-data/ts-1/wal/instance
I20260621 05:14:19.758575 3598 external_mini_cluster.cc:1428] Setting key 75b8e86213e02832e5eb8f20f213e1b5
I20260621 05:14:19.759086 21182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:14:19.754374 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.131.130:46633 (local address 127.3.131.190:38169)
0621 05:14:19.754452 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0621 05:14:19.754454 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:14:19.754686 (+ 232us) server_negotiation.cc:407] Connection header received
0621 05:14:19.755291 (+ 605us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:14:19.755293 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:14:19.755305 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:14:19.755360 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:14:19.756059 (+ 699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:14:19.756561 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:14:19.757550 (+ 989us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:14:19.757746 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:14:19.757820 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:14:19.758784 (+ 964us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:14:19.758786 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:14:19.758786 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 05:14:19.758791 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:14:19.758813 (+ 22us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 05:14:19.758817 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:14:19.758885 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0621 05:14:19.758999 (+ 114us) server_negotiation.cc:299] Negotiation successful
0621 05:14:19.759023 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260621 05:14:19.759378 21464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:14:19.754348 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.131.190:38169 (local address 127.3.131.130:46633)
0621 05:14:19.754555 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0621 05:14:19.754599 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0621 05:14:19.755130 (+ 531us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:14:19.755432 (+ 302us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:14:19.755436 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:14:19.755450 (+ 14us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:14:19.755957 (+ 507us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:14:19.755961 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:14:19.756657 (+ 696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:14:19.756658 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:14:19.757446 (+ 788us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:14:19.757448 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:14:19.757827 (+ 379us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:14:19.757828 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:14:19.757961 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:14:19.758617 (+ 656us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0621 05:14:19.758617 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:14:19.758667 (+ 50us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0621 05:14:19.758698 (+ 31us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 05:14:19.758915 (+ 217us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 05:14:19.758923 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 05:14:19.758926 (+ 3us) client_negotiation.cc:769] Sending connection context
0621 05:14:19.759004 (+ 78us) client_negotiation.cc:240] Negotiation successful
0621 05:14:19.759109 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":72,"threads_started":1}
I20260621 05:14:19.759706 3598 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWZOx2_/build/release/bin/kudu
/tmp/dist-test-taskWZOx2_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782018616648578-3598-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782018616648578-3598-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782018616648578-3598-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782018616648578-3598-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.131.131:0
--local_ip_for_outbound_sockets=127.3.131.131
--webserver_interface=127.3.131.131
--webserver_port=0
--tserver_master_addrs=127.3.131.190:38169
--builtin_ntp_servers=127.3.131.148:42409
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260621 05:14:19.759784 21462 heartbeater.cc:347] Connected to a master server at 127.3.131.190:38169
I20260621 05:14:19.759872 21462 heartbeater.cc:464] Registering TS with master...
I20260621 05:14:19.760042 21462 heartbeater.cc:511] Master 127.3.131.190:38169 requested a full tablet report, sending...
I20260621 05:14:19.760376 21143 ts_manager.cc:194] Registered new tserver with Master: 7faa5bd5726e4ae8a9ed344a6106f01f (127.3.131.130:39315)
I20260621 05:14:19.760746 21143 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.3.131.130:46633
W20260621 05:14:19.842294 21465 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 05:14:19.842516 21465 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 05:14:19.842541 21465 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 05:14:19.843837 21465 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 05:14:19.843886 21465 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 05:14:19.843914 21465 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.131.131
I20260621 05:14:19.845333 21465 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.131.148:42409
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782018616648578-3598-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782018616648578-3598-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.131.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782018616648578-3598-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.131.131
--webserver_port=0
--tserver_master_addrs=127.3.131.190:38169
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21465
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.131.131
--log_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782018616648578-3598-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 21 Jun 2026 04:16:16 UTC on ccycloud.quasar-xycaha.root.comops.site
build id 4457881
I20260621 05:14:19.845592 21465 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 05:14:19.845908 21465 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 05:14:19.851104 21473 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:14:19.851135 21471 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:14:19.851321 21465 server_base.cc:1034] running on GCE node
W20260621 05:14:19.851135 21470 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:14:19.851634 21465 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 05:14:19.851897 21465 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 05:14:19.853042 21465 hybrid_clock.cc:648] HybridClock initialized: now 1782018859853027 us; error 29 us; skew 500 ppm
I20260621 05:14:19.854452 21465 webserver.cc:460] Webserver started at http://127.3.131.131:43025/ using document root <none> and password file <none>
I20260621 05:14:19.854720 21465 fs_manager.cc:359] Metadata directory not provided
I20260621 05:14:19.854789 21465 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 05:14:19.854933 21465 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 05:14:19.856129 21465 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782018616648578-3598-0/minicluster-data/ts-2/data/instance:
uuid: "62f073c2c532410eb5684050f5f24a73"
format_stamp: "Formatted at 2026-06-21 05:14:19 on dist-test-slave-vrq1"
server_key: "882d8a888940526485547625990bc6bb"
server_key_iv: "34a5b093fb628d9e1e5ca679c11d0325"
server_key_version: "encryptionkey@0"
I20260621 05:14:19.856506 21465 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782018616648578-3598-0/minicluster-data/ts-2/wal/instance:
uuid: "62f073c2c532410eb5684050f5f24a73"
format_stamp: "Formatted at 2026-06-21 05:14:19 on dist-test-slave-vrq1"
server_key: "882d8a888940526485547625990bc6bb"
server_key_iv: "34a5b093fb628d9e1e5ca679c11d0325"
server_key_version: "encryptionkey@0"
I20260621 05:14:19.857775 21465 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260621 05:14:19.858464 21479 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 05:14:19.858628 21465 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260621 05:14:19.858726 21465 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782018616648578-3598-0/minicluster-data/ts-2/data,/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782018616648578-3598-0/minicluster-data/ts-2/wal
uuid: "62f073c2c532410eb5684050f5f24a73"
format_stamp: "Formatted at 2026-06-21 05:14:19 on dist-test-slave-vrq1"
server_key: "882d8a888940526485547625990bc6bb"
server_key_iv: "34a5b093fb628d9e1e5ca679c11d0325"
server_key_version: "encryptionkey@0"
I20260621 05:14:19.858803 21465 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782018616648578-3598-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782018616648578-3598-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782018616648578-3598-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 05:14:19.868346 21465 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 05:14:19.868692 21465 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 05:14:19.868856 21465 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 05:14:19.869095 21465 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 05:14:19.869437 21465 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 05:14:19.869488 21465 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 05:14:19.869529 21465 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 05:14:19.869554 21465 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 05:14:19.875641 21465 rpc_server.cc:307] RPC server started. Bound to: 127.3.131.131:33997
I20260621 05:14:19.875690 21592 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.131.131:33997 every 8 connection(s)
I20260621 05:14:19.876063 21465 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782018616648578-3598-0/minicluster-data/ts-2/data/info.pb
I20260621 05:14:19.881973 21182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:14:19.876880 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.131.131:48385 (local address 127.3.131.190:38169)
0621 05:14:19.876962 (+ 82us) server_negotiation.cc:206] Beginning negotiation
0621 05:14:19.876965 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0621 05:14:19.877208 (+ 243us) server_negotiation.cc:407] Connection header received
0621 05:14:19.877847 (+ 639us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:14:19.877848 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:14:19.877860 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:14:19.877910 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:14:19.878611 (+ 701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:14:19.879147 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:14:19.880270 (+ 1123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:14:19.880491 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:14:19.880556 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:14:19.881629 (+ 1073us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:14:19.881631 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:14:19.881631 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 05:14:19.881639 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:14:19.881659 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 05:14:19.881665 (+ 6us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:14:19.881722 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0621 05:14:19.881871 (+ 149us) server_negotiation.cc:299] Negotiation successful
0621 05:14:19.881901 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260621 05:14:19.882239 21595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:14:19.876853 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.131.190:38169 (local address 127.3.131.131:48385)
0621 05:14:19.877082 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0621 05:14:19.877122 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0621 05:14:19.877726 (+ 604us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:14:19.877973 (+ 247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:14:19.877977 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:14:19.877986 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:14:19.878513 (+ 527us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:14:19.878517 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:14:19.879298 (+ 781us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:14:19.879299 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:14:19.880163 (+ 864us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:14:19.880166 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:14:19.880619 (+ 453us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:14:19.880621 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:14:19.880783 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:14:19.881440 (+ 657us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0621 05:14:19.881441 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:14:19.881473 (+ 32us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0621 05:14:19.881499 (+ 26us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 05:14:19.881773 (+ 274us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 05:14:19.881779 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 05:14:19.881782 (+ 3us) client_negotiation.cc:769] Sending connection context
0621 05:14:19.881856 (+ 74us) client_negotiation.cc:240] Negotiation successful
0621 05:14:19.881961 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":165,"thread_start_us":122,"threads_started":1}
I20260621 05:14:19.883071 21593 heartbeater.cc:347] Connected to a master server at 127.3.131.190:38169
I20260621 05:14:19.883234 21593 heartbeater.cc:464] Registering TS with master...
I20260621 05:14:19.883442 21593 heartbeater.cc:511] Master 127.3.131.190:38169 requested a full tablet report, sending...
I20260621 05:14:19.883919 21143 ts_manager.cc:194] Registered new tserver with Master: 62f073c2c532410eb5684050f5f24a73 (127.3.131.131:33997)
I20260621 05:14:19.884310 21143 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.3.131.131:48385
I20260621 05:14:19.884866 3598 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWZOx2_/build/release/bin/kudu as pid 21465
I20260621 05:14:19.884961 3598 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782018616648578-3598-0/minicluster-data/ts-2/wal/instance
I20260621 05:14:19.885190 3598 external_mini_cluster.cc:1428] Setting key a207a0a2a36a784eaf7e5c0fb321ec91
I20260621 05:14:19.885890 3598 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260621 05:14:19.889823 21182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:14:19.887237 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33106 (local address 127.3.131.190:38169)
0621 05:14:19.887297 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0621 05:14:19.887299 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:14:19.887418 (+ 119us) server_negotiation.cc:407] Connection header received
0621 05:14:19.887462 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:14:19.887462 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:14:19.887471 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:14:19.887519 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:14:19.887649 (+ 130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:14:19.888122 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:14:19.888834 (+ 712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:14:19.889014 (+ 180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:14:19.889068 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:14:19.889645 (+ 577us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:14:19.889646 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:14:19.889647 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 05:14:19.889651 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:14:19.889665 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 05:14:19.889669 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:14:19.889707 (+ 38us) server_negotiation.cc:1035] Waiting for connection context
0621 05:14:19.889747 (+ 40us) server_negotiation.cc:299] Negotiation successful
0621 05:14:19.889766 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260621 05:14:19.891297 21143 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33106:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260621 05:14:19.892853 21143 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260621 05:14:19.899402 21605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:14:19.895175 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.131.131:33997 (local address 127.0.0.1:41908)
0621 05:14:19.895576 (+ 401us) negotiation.cc:107] Waiting for socket to connect
0621 05:14:19.895647 (+ 71us) client_negotiation.cc:174] Beginning negotiation
0621 05:14:19.895750 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:14:19.896097 (+ 347us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:14:19.896097 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:14:19.896099 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 05:14:19.896136 (+ 37us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:14:19.896138 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:14:19.896983 (+ 845us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:14:19.896984 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:14:19.898319 (+ 1335us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:14:19.898321 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:14:19.899031 (+ 710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:14:19.899031 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:14:19.899107 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:14:19.899120 (+ 13us) client_negotiation.cc:769] Sending connection context
0621 05:14:19.899238 (+ 118us) client_negotiation.cc:240] Negotiation successful
0621 05:14:19.899295 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":385,"mutex_wait_us":105,"thread_start_us":100,"threads_started":1}
I20260621 05:14:19.899509 21601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:14:19.894965 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.131.129:38477 (local address 127.0.0.1:58880)
0621 05:14:19.895282 (+ 317us) negotiation.cc:107] Waiting for socket to connect
0621 05:14:19.895317 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0621 05:14:19.895417 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:14:19.895734 (+ 317us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:14:19.895735 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:14:19.895738 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 05:14:19.895780 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:14:19.895782 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:14:19.897030 (+ 1248us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:14:19.897031 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:14:19.898317 (+ 1286us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:14:19.898320 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:14:19.899240 (+ 920us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:14:19.899241 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:14:19.899294 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:14:19.899297 (+ 3us) client_negotiation.cc:769] Sending connection context
0621 05:14:19.899355 (+ 58us) client_negotiation.cc:240] Negotiation successful
0621 05:14:19.899387 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":301,"mutex_wait_us":11,"thread_start_us":95,"threads_started":1}
I20260621 05:14:19.899722 21603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:14:19.895045 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58880 (local address 127.3.131.129:38477)
0621 05:14:19.895211 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0621 05:14:19.895215 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 05:14:19.895384 (+ 169us) server_negotiation.cc:407] Connection header received
0621 05:14:19.895534 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:14:19.895537 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:14:19.895552 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:14:19.895623 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 05:14:19.895845 (+ 222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:14:19.896952 (+ 1107us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:14:19.898400 (+ 1448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:14:19.899130 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:14:19.899219 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:14:19.899295 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0621 05:14:19.899371 (+ 76us) server_negotiation.cc:299] Negotiation successful
0621 05:14:19.899416 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":102,"thread_start_us":72,"threads_started":1}
I20260621 05:14:19.899739 21606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:14:19.895407 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41908 (local address 127.3.131.131:33997)
0621 05:14:19.895950 (+ 543us) server_negotiation.cc:206] Beginning negotiation
0621 05:14:19.895953 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0621 05:14:19.895961 (+ 8us) server_negotiation.cc:407] Connection header received
0621 05:14:19.895991 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:14:19.895992 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:14:19.896004 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:14:19.896084 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 05:14:19.896222 (+ 138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:14:19.896892 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:14:19.898426 (+ 1534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:14:19.898934 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:14:19.899019 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:14:19.899117 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0621 05:14:19.899617 (+ 500us) server_negotiation.cc:299] Negotiation successful
0621 05:14:19.899667 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":488,"thread_start_us":84,"threads_started":1}
I20260621 05:14:19.899953 21604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:14:19.895041 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46302 (local address 127.3.131.130:39315)
0621 05:14:19.895304 (+ 263us) server_negotiation.cc:206] Beginning negotiation
0621 05:14:19.895308 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 05:14:19.895356 (+ 48us) server_negotiation.cc:407] Connection header received
0621 05:14:19.895468 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:14:19.895470 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:14:19.895484 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:14:19.895555 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 05:14:19.895754 (+ 199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:14:19.896944 (+ 1190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:14:19.898311 (+ 1367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:14:19.899068 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:14:19.899146 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:14:19.899270 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0621 05:14:19.899649 (+ 379us) server_negotiation.cc:299] Negotiation successful
0621 05:14:19.899694 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":226,"thread_start_us":86,"threads_started":1}
I20260621 05:14:19.900206 21602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:14:19.894953 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.131.130:39315 (local address 127.0.0.1:46302)
0621 05:14:19.895249 (+ 296us) negotiation.cc:107] Waiting for socket to connect
0621 05:14:19.895304 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0621 05:14:19.895380 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:14:19.895573 (+ 193us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:14:19.895577 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:14:19.895579 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 05:14:19.895652 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:14:19.895654 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:14:19.897049 (+ 1395us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:14:19.897050 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:14:19.898215 (+ 1165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:14:19.898218 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:14:19.899309 (+ 1091us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:14:19.899310 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:14:19.899378 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:14:19.899382 (+ 4us) client_negotiation.cc:769] Sending connection context
0621 05:14:19.900089 (+ 707us) client_negotiation.cc:240] Negotiation successful
0621 05:14:19.900139 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":263,"mutex_wait_us":135,"thread_start_us":81,"threads_started":1}
I20260621 05:14:19.900439 21265 tablet_service.cc:1467] Processing CreateTablet for tablet 8a737d4622944ca6a95ac4e7cb0facd5 (DEFAULT_TABLE table=test-table [id=e2dbd3f1c2f94ae0b421be75c23cfd01]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 05:14:19.900437 21527 tablet_service.cc:1467] Processing CreateTablet for tablet 8a737d4622944ca6a95ac4e7cb0facd5 (DEFAULT_TABLE table=test-table [id=e2dbd3f1c2f94ae0b421be75c23cfd01]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 05:14:19.900743 21265 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8a737d4622944ca6a95ac4e7cb0facd5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 05:14:19.900767 21527 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8a737d4622944ca6a95ac4e7cb0facd5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 05:14:19.903636 21607 tablet_bootstrap.cc:492] T 8a737d4622944ca6a95ac4e7cb0facd5 P 04f53e20f5cb4d6c9250fe7c4945913d: Bootstrap starting.
I20260621 05:14:19.904282 21607 tablet_bootstrap.cc:654] T 8a737d4622944ca6a95ac4e7cb0facd5 P 04f53e20f5cb4d6c9250fe7c4945913d: Neither blocks nor log segments found. Creating new log.
I20260621 05:14:19.904284 21396 tablet_service.cc:1467] Processing CreateTablet for tablet 8a737d4622944ca6a95ac4e7cb0facd5 (DEFAULT_TABLE table=test-table [id=e2dbd3f1c2f94ae0b421be75c23cfd01]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 05:14:19.904654 21396 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8a737d4622944ca6a95ac4e7cb0facd5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 05:14:19.904660 21607 log.cc:824] T 8a737d4622944ca6a95ac4e7cb0facd5 P 04f53e20f5cb4d6c9250fe7c4945913d: Log is configured to *not* fsync() on all Append() calls
I20260621 05:14:19.905031 21608 tablet_bootstrap.cc:492] T 8a737d4622944ca6a95ac4e7cb0facd5 P 62f073c2c532410eb5684050f5f24a73: Bootstrap starting.
I20260621 05:14:19.905484 21607 tablet_bootstrap.cc:492] T 8a737d4622944ca6a95ac4e7cb0facd5 P 04f53e20f5cb4d6c9250fe7c4945913d: No bootstrap required, opened a new log
I20260621 05:14:19.905557 21607 ts_tablet_manager.cc:1397] T 8a737d4622944ca6a95ac4e7cb0facd5 P 04f53e20f5cb4d6c9250fe7c4945913d: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260621 05:14:19.905725 21608 tablet_bootstrap.cc:654] T 8a737d4622944ca6a95ac4e7cb0facd5 P 62f073c2c532410eb5684050f5f24a73: Neither blocks nor log segments found. Creating new log.
I20260621 05:14:19.906050 21608 log.cc:824] T 8a737d4622944ca6a95ac4e7cb0facd5 P 62f073c2c532410eb5684050f5f24a73: Log is configured to *not* fsync() on all Append() calls
I20260621 05:14:19.906884 21608 tablet_bootstrap.cc:492] T 8a737d4622944ca6a95ac4e7cb0facd5 P 62f073c2c532410eb5684050f5f24a73: No bootstrap required, opened a new log
I20260621 05:14:19.906956 21608 ts_tablet_manager.cc:1397] T 8a737d4622944ca6a95ac4e7cb0facd5 P 62f073c2c532410eb5684050f5f24a73: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260621 05:14:19.908301 21607 raft_consensus.cc:348] T 8a737d4622944ca6a95ac4e7cb0facd5 P 04f53e20f5cb4d6c9250fe7c4945913d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7faa5bd5726e4ae8a9ed344a6106f01f" member_type: VOTER last_known_addr { host: "127.3.131.130" port: 39315 } } peers { permanent_uuid: "04f53e20f5cb4d6c9250fe7c4945913d" member_type: VOTER last_known_addr { host: "127.3.131.129" port: 38477 } } peers { permanent_uuid: "62f073c2c532410eb5684050f5f24a73" member_type: VOTER last_known_addr { host: "127.3.131.131" port: 33997 } }
I20260621 05:14:19.908480 21607 raft_consensus.cc:374] T 8a737d4622944ca6a95ac4e7cb0facd5 P 04f53e20f5cb4d6c9250fe7c4945913d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 05:14:19.908517 21607 raft_consensus.cc:729] T 8a737d4622944ca6a95ac4e7cb0facd5 P 04f53e20f5cb4d6c9250fe7c4945913d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 04f53e20f5cb4d6c9250fe7c4945913d, State: Initialized, Role: FOLLOWER
I20260621 05:14:19.908650 21607 consensus_queue.cc:260] T 8a737d4622944ca6a95ac4e7cb0facd5 P 04f53e20f5cb4d6c9250fe7c4945913d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7faa5bd5726e4ae8a9ed344a6106f01f" member_type: VOTER last_known_addr { host: "127.3.131.130" port: 39315 } } peers { permanent_uuid: "04f53e20f5cb4d6c9250fe7c4945913d" member_type: VOTER last_known_addr { host: "127.3.131.129" port: 38477 } } peers { permanent_uuid: "62f073c2c532410eb5684050f5f24a73" member_type: VOTER last_known_addr { host: "127.3.131.131" port: 33997 } }
I20260621 05:14:19.908926 21607 ts_tablet_manager.cc:1428] T 8a737d4622944ca6a95ac4e7cb0facd5 P 04f53e20f5cb4d6c9250fe7c4945913d: Time spent starting tablet: real 0.003s user 0.004s sys 0.000s
I20260621 05:14:19.908982 21331 heartbeater.cc:503] Master 127.3.131.190:38169 was elected leader, sending a full tablet report...
I20260621 05:14:19.909731 21608 raft_consensus.cc:348] T 8a737d4622944ca6a95ac4e7cb0facd5 P 62f073c2c532410eb5684050f5f24a73 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7faa5bd5726e4ae8a9ed344a6106f01f" member_type: VOTER last_known_addr { host: "127.3.131.130" port: 39315 } } peers { permanent_uuid: "04f53e20f5cb4d6c9250fe7c4945913d" member_type: VOTER last_known_addr { host: "127.3.131.129" port: 38477 } } peers { permanent_uuid: "62f073c2c532410eb5684050f5f24a73" member_type: VOTER last_known_addr { host: "127.3.131.131" port: 33997 } }
I20260621 05:14:19.909909 21608 raft_consensus.cc:374] T 8a737d4622944ca6a95ac4e7cb0facd5 P 62f073c2c532410eb5684050f5f24a73 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 05:14:19.909961 21608 raft_consensus.cc:729] T 8a737d4622944ca6a95ac4e7cb0facd5 P 62f073c2c532410eb5684050f5f24a73 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 62f073c2c532410eb5684050f5f24a73, State: Initialized, Role: FOLLOWER
I20260621 05:14:19.910082 21608 consensus_queue.cc:260] T 8a737d4622944ca6a95ac4e7cb0facd5 P 62f073c2c532410eb5684050f5f24a73 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7faa5bd5726e4ae8a9ed344a6106f01f" member_type: VOTER last_known_addr { host: "127.3.131.130" port: 39315 } } peers { permanent_uuid: "04f53e20f5cb4d6c9250fe7c4945913d" member_type: VOTER last_known_addr { host: "127.3.131.129" port: 38477 } } peers { permanent_uuid: "62f073c2c532410eb5684050f5f24a73" member_type: VOTER last_known_addr { host: "127.3.131.131" port: 33997 } }
I20260621 05:14:19.910377 21608 ts_tablet_manager.cc:1428] T 8a737d4622944ca6a95ac4e7cb0facd5 P 62f073c2c532410eb5684050f5f24a73: Time spent starting tablet: real 0.003s user 0.004s sys 0.000s
I20260621 05:14:19.910398 21593 heartbeater.cc:503] Master 127.3.131.190:38169 was elected leader, sending a full tablet report...
I20260621 05:14:19.910619 21611 tablet_bootstrap.cc:492] T 8a737d4622944ca6a95ac4e7cb0facd5 P 7faa5bd5726e4ae8a9ed344a6106f01f: Bootstrap starting.
I20260621 05:14:19.911367 21611 tablet_bootstrap.cc:654] T 8a737d4622944ca6a95ac4e7cb0facd5 P 7faa5bd5726e4ae8a9ed344a6106f01f: Neither blocks nor log segments found. Creating new log.
I20260621 05:14:19.911692 21611 log.cc:824] T 8a737d4622944ca6a95ac4e7cb0facd5 P 7faa5bd5726e4ae8a9ed344a6106f01f: Log is configured to *not* fsync() on all Append() calls
I20260621 05:14:19.912372 21611 tablet_bootstrap.cc:492] T 8a737d4622944ca6a95ac4e7cb0facd5 P 7faa5bd5726e4ae8a9ed344a6106f01f: No bootstrap required, opened a new log
I20260621 05:14:19.912428 21611 ts_tablet_manager.cc:1397] T 8a737d4622944ca6a95ac4e7cb0facd5 P 7faa5bd5726e4ae8a9ed344a6106f01f: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260621 05:14:19.914458 21611 raft_consensus.cc:348] T 8a737d4622944ca6a95ac4e7cb0facd5 P 7faa5bd5726e4ae8a9ed344a6106f01f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7faa5bd5726e4ae8a9ed344a6106f01f" member_type: VOTER last_known_addr { host: "127.3.131.130" port: 39315 } } peers { permanent_uuid: "04f53e20f5cb4d6c9250fe7c4945913d" member_type: VOTER last_known_addr { host: "127.3.131.129" port: 38477 } } peers { permanent_uuid: "62f073c2c532410eb5684050f5f24a73" member_type: VOTER last_known_addr { host: "127.3.131.131" port: 33997 } }
I20260621 05:14:19.914599 21611 raft_consensus.cc:374] T 8a737d4622944ca6a95ac4e7cb0facd5 P 7faa5bd5726e4ae8a9ed344a6106f01f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 05:14:19.914623 21611 raft_consensus.cc:729] T 8a737d4622944ca6a95ac4e7cb0facd5 P 7faa5bd5726e4ae8a9ed344a6106f01f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7faa5bd5726e4ae8a9ed344a6106f01f, State: Initialized, Role: FOLLOWER
I20260621 05:14:19.914783 21611 consensus_queue.cc:260] T 8a737d4622944ca6a95ac4e7cb0facd5 P 7faa5bd5726e4ae8a9ed344a6106f01f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7faa5bd5726e4ae8a9ed344a6106f01f" member_type: VOTER last_known_addr { host: "127.3.131.130" port: 39315 } } peers { permanent_uuid: "04f53e20f5cb4d6c9250fe7c4945913d" member_type: VOTER last_known_addr { host: "127.3.131.129" port: 38477 } } peers { permanent_uuid: "62f073c2c532410eb5684050f5f24a73" member_type: VOTER last_known_addr { host: "127.3.131.131" port: 33997 } }
I20260621 05:14:19.915053 21611 ts_tablet_manager.cc:1428] T 8a737d4622944ca6a95ac4e7cb0facd5 P 7faa5bd5726e4ae8a9ed344a6106f01f: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260621 05:14:19.915112 21462 heartbeater.cc:503] Master 127.3.131.190:38169 was elected leader, sending a full tablet report...
I20260621 05:14:19.952375 21612 raft_consensus.cc:482] T 8a737d4622944ca6a95ac4e7cb0facd5 P 04f53e20f5cb4d6c9250fe7c4945913d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260621 05:14:19.952520 21612 raft_consensus.cc:504] T 8a737d4622944ca6a95ac4e7cb0facd5 P 04f53e20f5cb4d6c9250fe7c4945913d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7faa5bd5726e4ae8a9ed344a6106f01f" member_type: VOTER last_known_addr { host: "127.3.131.130" port: 39315 } } peers { permanent_uuid: "04f53e20f5cb4d6c9250fe7c4945913d" member_type: VOTER last_known_addr { host: "127.3.131.129" port: 38477 } } peers { permanent_uuid: "62f073c2c532410eb5684050f5f24a73" member_type: VOTER last_known_addr { host: "127.3.131.131" port: 33997 } }
I20260621 05:14:19.952965 21612 leader_election.cc:288] T 8a737d4622944ca6a95ac4e7cb0facd5 P 04f53e20f5cb4d6c9250fe7c4945913d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7faa5bd5726e4ae8a9ed344a6106f01f (127.3.131.130:39315), 62f073c2c532410eb5684050f5f24a73 (127.3.131.131:33997)
I20260621 05:14:19.956260 21604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:14:19.953311 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.131.129:46099 (local address 127.3.131.130:39315)
0621 05:14:19.953379 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0621 05:14:19.953381 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:14:19.953391 (+ 10us) server_negotiation.cc:407] Connection header received
0621 05:14:19.953448 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:14:19.953449 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:14:19.953459 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:14:19.953517 (+ 58us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 05:14:19.953710 (+ 193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:14:19.954282 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:14:19.955510 (+ 1228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:14:19.955908 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:14:19.955997 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:14:19.956006 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0621 05:14:19.956155 (+ 149us) server_negotiation.cc:299] Negotiation successful
0621 05:14:19.956198 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260621 05:14:19.956260 21333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:14:19.953225 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.131.130:39315 (local address 127.3.131.129:46099)
0621 05:14:19.953292 (+ 67us) negotiation.cc:107] Waiting for socket to connect
0621 05:14:19.953306 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0621 05:14:19.953375 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:14:19.953525 (+ 150us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:14:19.953526 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:14:19.953527 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 05:14:19.953617 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:14:19.953620 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:14:19.954378 (+ 758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:14:19.954379 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:14:19.955415 (+ 1036us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:14:19.955418 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:14:19.955995 (+ 577us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:14:19.955996 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:14:19.956067 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:14:19.956070 (+ 3us) client_negotiation.cc:769] Sending connection context
0621 05:14:19.956129 (+ 59us) client_negotiation.cc:240] Negotiation successful
0621 05:14:19.956177 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":42,"mutex_wait_us":61}
I20260621 05:14:19.956396 21616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:14:19.953350 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.131.131:33997 (local address 127.3.131.129:40045)
0621 05:14:19.953540 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0621 05:14:19.953555 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0621 05:14:19.953614 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:14:19.953749 (+ 135us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:14:19.953750 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:14:19.953752 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 05:14:19.953815 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:14:19.953817 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:14:19.954582 (+ 765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:14:19.954583 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:14:19.955502 (+ 919us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:14:19.955505 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:14:19.956059 (+ 554us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:14:19.956060 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:14:19.956137 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:14:19.956140 (+ 3us) client_negotiation.cc:769] Sending connection context
0621 05:14:19.956197 (+ 57us) client_negotiation.cc:240] Negotiation successful
0621 05:14:19.956237 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":170,"thread_start_us":136,"threads_started":1}
I20260621 05:14:19.956636 21606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:14:19.953427 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.131.129:40045 (local address 127.3.131.131:33997)
0621 05:14:19.953487 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0621 05:14:19.953489 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:14:19.953618 (+ 129us) server_negotiation.cc:407] Connection header received
0621 05:14:19.953669 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:14:19.953671 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:14:19.953681 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:14:19.953739 (+ 58us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 05:14:19.953877 (+ 138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:14:19.954505 (+ 628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:14:19.955589 (+ 1084us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:14:19.955966 (+ 377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:14:19.956058 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:14:19.956066 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0621 05:14:19.956224 (+ 158us) server_negotiation.cc:299] Negotiation successful
0621 05:14:19.956265 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260621 05:14:19.956776 21416 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a737d4622944ca6a95ac4e7cb0facd5" candidate_uuid: "04f53e20f5cb4d6c9250fe7c4945913d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7faa5bd5726e4ae8a9ed344a6106f01f" is_pre_election: true
I20260621 05:14:19.956972 21416 raft_consensus.cc:2454] T 8a737d4622944ca6a95ac4e7cb0facd5 P 7faa5bd5726e4ae8a9ed344a6106f01f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 04f53e20f5cb4d6c9250fe7c4945913d in term 0.
I20260621 05:14:19.956938 21547 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a737d4622944ca6a95ac4e7cb0facd5" candidate_uuid: "04f53e20f5cb4d6c9250fe7c4945913d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "62f073c2c532410eb5684050f5f24a73" is_pre_election: true
I20260621 05:14:19.957090 21547 raft_consensus.cc:2454] T 8a737d4622944ca6a95ac4e7cb0facd5 P 62f073c2c532410eb5684050f5f24a73 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 04f53e20f5cb4d6c9250fe7c4945913d in term 0.
I20260621 05:14:19.957223 21219 leader_election.cc:302] T 8a737d4622944ca6a95ac4e7cb0facd5 P 04f53e20f5cb4d6c9250fe7c4945913d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 04f53e20f5cb4d6c9250fe7c4945913d, 7faa5bd5726e4ae8a9ed344a6106f01f; no voters:
I20260621 05:14:19.957368 21612 raft_consensus.cc:2781] T 8a737d4622944ca6a95ac4e7cb0facd5 P 04f53e20f5cb4d6c9250fe7c4945913d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260621 05:14:19.957424 21612 raft_consensus.cc:482] T 8a737d4622944ca6a95ac4e7cb0facd5 P 04f53e20f5cb4d6c9250fe7c4945913d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260621 05:14:19.957453 21612 raft_consensus.cc:3037] T 8a737d4622944ca6a95ac4e7cb0facd5 P 04f53e20f5cb4d6c9250fe7c4945913d [term 0 FOLLOWER]: Advancing to term 1
I20260621 05:14:19.958159 21612 raft_consensus.cc:504] T 8a737d4622944ca6a95ac4e7cb0facd5 P 04f53e20f5cb4d6c9250fe7c4945913d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7faa5bd5726e4ae8a9ed344a6106f01f" member_type: VOTER last_known_addr { host: "127.3.131.130" port: 39315 } } peers { permanent_uuid: "04f53e20f5cb4d6c9250fe7c4945913d" member_type: VOTER last_known_addr { host: "127.3.131.129" port: 38477 } } peers { permanent_uuid: "62f073c2c532410eb5684050f5f24a73" member_type: VOTER last_known_addr { host: "127.3.131.131" port: 33997 } }
I20260621 05:14:19.958293 21612 leader_election.cc:288] T 8a737d4622944ca6a95ac4e7cb0facd5 P 04f53e20f5cb4d6c9250fe7c4945913d [CANDIDATE]: Term 1 election: Requested vote from peers 7faa5bd5726e4ae8a9ed344a6106f01f (127.3.131.130:39315), 62f073c2c532410eb5684050f5f24a73 (127.3.131.131:33997)
I20260621 05:14:19.958437 21416 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a737d4622944ca6a95ac4e7cb0facd5" candidate_uuid: "04f53e20f5cb4d6c9250fe7c4945913d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7faa5bd5726e4ae8a9ed344a6106f01f"
I20260621 05:14:19.958484 21547 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a737d4622944ca6a95ac4e7cb0facd5" candidate_uuid: "04f53e20f5cb4d6c9250fe7c4945913d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "62f073c2c532410eb5684050f5f24a73"
I20260621 05:14:19.958519 21416 raft_consensus.cc:3037] T 8a737d4622944ca6a95ac4e7cb0facd5 P 7faa5bd5726e4ae8a9ed344a6106f01f [term 0 FOLLOWER]: Advancing to term 1
I20260621 05:14:19.958577 21547 raft_consensus.cc:3037] T 8a737d4622944ca6a95ac4e7cb0facd5 P 62f073c2c532410eb5684050f5f24a73 [term 0 FOLLOWER]: Advancing to term 1
I20260621 05:14:19.959313 21416 raft_consensus.cc:2454] T 8a737d4622944ca6a95ac4e7cb0facd5 P 7faa5bd5726e4ae8a9ed344a6106f01f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 04f53e20f5cb4d6c9250fe7c4945913d in term 1.
I20260621 05:14:19.959429 21547 raft_consensus.cc:2454] T 8a737d4622944ca6a95ac4e7cb0facd5 P 62f073c2c532410eb5684050f5f24a73 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 04f53e20f5cb4d6c9250fe7c4945913d in term 1.
I20260621 05:14:19.959473 21219 leader_election.cc:302] T 8a737d4622944ca6a95ac4e7cb0facd5 P 04f53e20f5cb4d6c9250fe7c4945913d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 04f53e20f5cb4d6c9250fe7c4945913d, 7faa5bd5726e4ae8a9ed344a6106f01f; no voters:
I20260621 05:14:19.959573 21612 raft_consensus.cc:2781] T 8a737d4622944ca6a95ac4e7cb0facd5 P 04f53e20f5cb4d6c9250fe7c4945913d [term 1 FOLLOWER]: Leader election won for term 1
I20260621 05:14:19.959760 21612 raft_consensus.cc:686] T 8a737d4622944ca6a95ac4e7cb0facd5 P 04f53e20f5cb4d6c9250fe7c4945913d [term 1 LEADER]: Becoming Leader. State: Replica: 04f53e20f5cb4d6c9250fe7c4945913d, State: Running, Role: LEADER
I20260621 05:14:19.959882 21612 consensus_queue.cc:237] T 8a737d4622944ca6a95ac4e7cb0facd5 P 04f53e20f5cb4d6c9250fe7c4945913d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7faa5bd5726e4ae8a9ed344a6106f01f" member_type: VOTER last_known_addr { host: "127.3.131.130" port: 39315 } } peers { permanent_uuid: "04f53e20f5cb4d6c9250fe7c4945913d" member_type: VOTER last_known_addr { host: "127.3.131.129" port: 38477 } } peers { permanent_uuid: "62f073c2c532410eb5684050f5f24a73" member_type: VOTER last_known_addr { host: "127.3.131.131" port: 33997 } }
I20260621 05:14:19.960732 21143 catalog_manager.cc:5495] T 8a737d4622944ca6a95ac4e7cb0facd5 P 04f53e20f5cb4d6c9250fe7c4945913d reported cstate change: term changed from 0 to 1, leader changed from <none> to 04f53e20f5cb4d6c9250fe7c4945913d (127.3.131.129). New cstate: current_term: 1 leader_uuid: "04f53e20f5cb4d6c9250fe7c4945913d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7faa5bd5726e4ae8a9ed344a6106f01f" member_type: VOTER last_known_addr { host: "127.3.131.130" port: 39315 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "04f53e20f5cb4d6c9250fe7c4945913d" member_type: VOTER last_known_addr { host: "127.3.131.129" port: 38477 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "62f073c2c532410eb5684050f5f24a73" member_type: VOTER last_known_addr { host: "127.3.131.131" port: 33997 } health_report { overall_health: UNKNOWN } } }
I20260621 05:14:19.975111 21603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:14:19.972170 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58888 (local address 127.3.131.129:38477)
0621 05:14:19.972234 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0621 05:14:19.972237 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0621 05:14:19.972245 (+ 8us) server_negotiation.cc:407] Connection header received
0621 05:14:19.972287 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:14:19.972288 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:14:19.972315 (+ 27us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:14:19.972390 (+ 75us) server_negotiation.cc:226] Negotiated authn=TOKEN
0621 05:14:19.972523 (+ 133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:14:19.973009 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:14:19.973944 (+ 935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:14:19.974265 (+ 321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:14:19.974343 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:14:19.974512 (+ 169us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0621 05:14:19.974692 (+ 180us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0621 05:14:19.974777 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0621 05:14:19.974965 (+ 188us) server_negotiation.cc:299] Negotiation successful
0621 05:14:19.974992 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260621 05:14:19.977710 21416 raft_consensus.cc:1261] T 8a737d4622944ca6a95ac4e7cb0facd5 P 7faa5bd5726e4ae8a9ed344a6106f01f [term 1 FOLLOWER]: Refusing update from remote peer 04f53e20f5cb4d6c9250fe7c4945913d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 05:14:19.978091 21547 raft_consensus.cc:1261] T 8a737d4622944ca6a95ac4e7cb0facd5 P 62f073c2c532410eb5684050f5f24a73 [term 1 FOLLOWER]: Refusing update from remote peer 04f53e20f5cb4d6c9250fe7c4945913d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 05:14:19.978395 21612 consensus_queue.cc:1035] T 8a737d4622944ca6a95ac4e7cb0facd5 P 04f53e20f5cb4d6c9250fe7c4945913d [LEADER]: Connected to new peer: Peer: permanent_uuid: "7faa5bd5726e4ae8a9ed344a6106f01f" member_type: VOTER last_known_addr { host: "127.3.131.130" port: 39315 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 05:14:19.978600 21612 consensus_queue.cc:1035] T 8a737d4622944ca6a95ac4e7cb0facd5 P 04f53e20f5cb4d6c9250fe7c4945913d [LEADER]: Connected to new peer: Peer: permanent_uuid: "62f073c2c532410eb5684050f5f24a73" member_type: VOTER last_known_addr { host: "127.3.131.131" port: 33997 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 05:14:19.981107 21620 mvcc.cc:205] Tried to move back new op lower bound from 7299149250465103872 to 7299149250396372992. Current Snapshot: MvccSnapshot[applied={T|T < 7299149250465103872}]
I20260621 05:14:19.984105 21623 mvcc.cc:205] Tried to move back new op lower bound from 7299149250465103872 to 7299149250396372992. Current Snapshot: MvccSnapshot[applied={T|T < 7299149250465103872 or (T in {7299149250465103872})}]
I20260621 05:14:19.984742 21143 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33106:
table { table_name: "test-table" } modify_external_catalogs: true
I20260621 05:14:19.984831 21143 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33106:
table { table_name: "test-table" } modify_external_catalogs: true
I20260621 05:14:19.985634 21143 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 07cafeb58da04f02806c39abc67c24f1: Sending DeleteTablet for 3 replicas of tablet 8a737d4622944ca6a95ac4e7cb0facd5
I20260621 05:14:19.985985 21265 tablet_service.cc:1514] Processing DeleteTablet for tablet 8a737d4622944ca6a95ac4e7cb0facd5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 05:14:19 UTC) from {username='slave'} at 127.0.0.1:58880
I20260621 05:14:19.985993 21396 tablet_service.cc:1514] Processing DeleteTablet for tablet 8a737d4622944ca6a95ac4e7cb0facd5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 05:14:19 UTC) from {username='slave'} at 127.0.0.1:46302
I20260621 05:14:19.986048 21527 tablet_service.cc:1514] Processing DeleteTablet for tablet 8a737d4622944ca6a95ac4e7cb0facd5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 05:14:19 UTC) from {username='slave'} at 127.0.0.1:41908
I20260621 05:14:19.986404 21629 tablet_replica.cc:330] T 8a737d4622944ca6a95ac4e7cb0facd5 P 7faa5bd5726e4ae8a9ed344a6106f01f: stopping tablet replica
I20260621 05:14:19.986454 21631 tablet_replica.cc:330] T 8a737d4622944ca6a95ac4e7cb0facd5 P 62f073c2c532410eb5684050f5f24a73: stopping tablet replica
I20260621 05:14:19.986531 21629 raft_consensus.cc:2229] T 8a737d4622944ca6a95ac4e7cb0facd5 P 7faa5bd5726e4ae8a9ed344a6106f01f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260621 05:14:19.986538 21631 raft_consensus.cc:2229] T 8a737d4622944ca6a95ac4e7cb0facd5 P 62f073c2c532410eb5684050f5f24a73 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260621 05:14:19.986644 21629 raft_consensus.cc:2258] T 8a737d4622944ca6a95ac4e7cb0facd5 P 7faa5bd5726e4ae8a9ed344a6106f01f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 05:14:19.986644 21631 raft_consensus.cc:2258] T 8a737d4622944ca6a95ac4e7cb0facd5 P 62f073c2c532410eb5684050f5f24a73 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 05:14:19.986826 3598 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWZOx2_/build/release/bin/kudu with pid 21203
I20260621 05:14:19.988106 21631 ts_tablet_manager.cc:1905] T 8a737d4622944ca6a95ac4e7cb0facd5 P 62f073c2c532410eb5684050f5f24a73: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 05:14:19.989265 21629 ts_tablet_manager.cc:1905] T 8a737d4622944ca6a95ac4e7cb0facd5 P 7faa5bd5726e4ae8a9ed344a6106f01f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 05:14:19.989676 21631 ts_tablet_manager.cc:1918] T 8a737d4622944ca6a95ac4e7cb0facd5 P 62f073c2c532410eb5684050f5f24a73: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260621 05:14:19.989768 21631 log.cc:1196] T 8a737d4622944ca6a95ac4e7cb0facd5 P 62f073c2c532410eb5684050f5f24a73: Deleting WAL directory at /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782018616648578-3598-0/minicluster-data/ts-2/wal/wals/8a737d4622944ca6a95ac4e7cb0facd5
I20260621 05:14:19.990029 21631 ts_tablet_manager.cc:1939] T 8a737d4622944ca6a95ac4e7cb0facd5 P 62f073c2c532410eb5684050f5f24a73: Deleting consensus metadata
I20260621 05:14:19.990561 21128 catalog_manager.cc:4841] TS 62f073c2c532410eb5684050f5f24a73 (127.3.131.131:33997): tablet 8a737d4622944ca6a95ac4e7cb0facd5 (table test-table [id=e2dbd3f1c2f94ae0b421be75c23cfd01]) successfully deleted
I20260621 05:14:19.991254 21629 ts_tablet_manager.cc:1918] T 8a737d4622944ca6a95ac4e7cb0facd5 P 7faa5bd5726e4ae8a9ed344a6106f01f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260621 05:14:19.991335 21629 log.cc:1196] T 8a737d4622944ca6a95ac4e7cb0facd5 P 7faa5bd5726e4ae8a9ed344a6106f01f: Deleting WAL directory at /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782018616648578-3598-0/minicluster-data/ts-1/wal/wals/8a737d4622944ca6a95ac4e7cb0facd5
I20260621 05:14:19.991586 21629 ts_tablet_manager.cc:1939] T 8a737d4622944ca6a95ac4e7cb0facd5 P 7faa5bd5726e4ae8a9ed344a6106f01f: Deleting consensus metadata
I20260621 05:14:19.992051 21128 catalog_manager.cc:4841] TS 7faa5bd5726e4ae8a9ed344a6106f01f (127.3.131.130:39315): tablet 8a737d4622944ca6a95ac4e7cb0facd5 (table test-table [id=e2dbd3f1c2f94ae0b421be75c23cfd01]) successfully deleted
W20260621 05:14:19.993975 21130 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.3.131.129:38477 (error 108)
I20260621 05:14:19.994621 3598 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWZOx2_/build/release/bin/kudu with pid 21334
I20260621 05:14:19.994654 21602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:14:19.994429 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.131.129:38477 (local address 127.0.0.1:58898)
0621 05:14:19.994546 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0621 05:14:19.994611 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.131.129:38477: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
W20260621 05:14:19.995324 21130 catalog_manager.cc:4584] TS 04f53e20f5cb4d6c9250fe7c4945913d (127.3.131.129:38477): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 8a737d4622944ca6a95ac4e7cb0facd5: Network error: Client connection negotiation failed: client connection to 127.3.131.129:38477: connect: Connection refused (error 111)
I20260621 05:14:20.001062 3598 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWZOx2_/build/release/bin/kudu with pid 21465
I20260621 05:14:20.006101 3598 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWZOx2_/build/release/bin/kudu with pid 21112
2026-06-21T05:14:20Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (701 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782018616648578-3598-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 21 05:14:20 dist-test-slave-vrq1 krb5kdc[21635](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 21 05:14:20 dist-test-slave-vrq1 krb5kdc[21635](info): set up 2 sockets
Jun 21 05:14:20 dist-test-slave-vrq1 krb5kdc[21635](info): commencing operation
krb5kdc: starting...
W20260621 05:14:21.055366 3598 mini_kdc.cc:121] Time spent starting KDC: real 1.029s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782018616648578-3598-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782018616648578-3598-0/krb5kdc/test-user.keytab.
Jun 21 05:14:21 dist-test-slave-vrq1 krb5kdc[21635](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782018861, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-21T05:14:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-21T05:14:21Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.131.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.131.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.131.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782018616648578-3598-0/krb5kdc/kudu_127.3.131.190.keytab.
Entry for principal kudu/127.3.131.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782018616648578-3598-0/krb5kdc/kudu_127.3.131.190.keytab.
I20260621 05:14:21.191303 3598 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWZOx2_/build/release/bin/kudu
/tmp/dist-test-taskWZOx2_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782018616648578-3598-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782018616648578-3598-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782018616648578-3598-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782018616648578-3598-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.131.190:32797
--webserver_interface=127.3.131.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782018616648578-3598-0/krb5kdc/kudu_127.3.131.190.keytab
--principal=kudu/127.3.131.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.131.148:40107
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.131.190:32797
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782018616648578-3598-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782018616648578-3598-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782018616648578-3598-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 05:14:21.272826 21648 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 05:14:21.273088 21648 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 05:14:21.273126 21648 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 05:14:21.274358 21648 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 05:14:21.274436 21648 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 05:14:21.274461 21648 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 05:14:21.274482 21648 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 05:14:21.274510 21648 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 05:14:21.276113 21648 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.131.148:40107
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782018616648578-3598-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782018616648578-3598-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.131.190:32797
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782018616648578-3598-0/krb5kdc/kudu_127.3.131.190.keytab
--principal=kudu/127.3.131.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.131.190:32797
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782018616648578-3598-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.131.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21648
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782018616648578-3598-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 21 Jun 2026 04:16:16 UTC on ccycloud.quasar-xycaha.root.comops.site
build id 4457881
I20260621 05:14:21.276408 21648 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 05:14:21.276647 21648 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260621 05:14:21.281970 21648 server_base.cc:1034] running on GCE node
W20260621 05:14:21.281936 21656 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:14:21.281936 21654 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:14:21.281939 21653 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:14:21.282429 21648 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 05:14:21.282774 21648 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 05:14:21.283962 21648 hybrid_clock.cc:648] HybridClock initialized: now 1782018861283958 us; error 42 us; skew 500 ppm
Jun 21 05:14:21 dist-test-slave-vrq1 krb5kdc[21635](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782018861, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.131.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 05:14:21.292265 21648 init.cc:377] Logged in from keytab as kudu/127.3.131.190@KRBTEST.COM (short username kudu)
I20260621 05:14:21.292855 21648 webserver.cc:460] Webserver started at http://127.3.131.190:43593/ using document root <none> and password file <none>
I20260621 05:14:21.293089 21648 fs_manager.cc:359] Metadata directory not provided
I20260621 05:14:21.293159 21648 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 05:14:21.293314 21648 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 05:14:21.294476 21648 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782018616648578-3598-0/minicluster-data/master-0/data/instance:
uuid: "65b67b67bb4e44dea10c37f4dd2af5ec"
format_stamp: "Formatted at 2026-06-21 05:14:21 on dist-test-slave-vrq1"
server_key: "6cf9bfd14900dad9e2e33154404b5973"
server_key_iv: "cec0f3c19e873e8be78598f18db8031f"
server_key_version: "encryptionkey@0"
I20260621 05:14:21.294919 21648 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782018616648578-3598-0/minicluster-data/master-0/wal/instance:
uuid: "65b67b67bb4e44dea10c37f4dd2af5ec"
format_stamp: "Formatted at 2026-06-21 05:14:21 on dist-test-slave-vrq1"
server_key: "6cf9bfd14900dad9e2e33154404b5973"
server_key_iv: "cec0f3c19e873e8be78598f18db8031f"
server_key_version: "encryptionkey@0"
I20260621 05:14:21.297022 21648 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.002s sys 0.002s
I20260621 05:14:21.298061 21663 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 05:14:21.298285 21648 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260621 05:14:21.298383 21648 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782018616648578-3598-0/minicluster-data/master-0/data,/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782018616648578-3598-0/minicluster-data/master-0/wal
uuid: "65b67b67bb4e44dea10c37f4dd2af5ec"
format_stamp: "Formatted at 2026-06-21 05:14:21 on dist-test-slave-vrq1"
server_key: "6cf9bfd14900dad9e2e33154404b5973"
server_key_iv: "cec0f3c19e873e8be78598f18db8031f"
server_key_version: "encryptionkey@0"
I20260621 05:14:21.298463 21648 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782018616648578-3598-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782018616648578-3598-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782018616648578-3598-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 05:14:21.310536 21648 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 05:14:21.312892 21648 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 05:14:21.313056 21648 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 05:14:21.317137 21648 rpc_server.cc:307] RPC server started. Bound to: 127.3.131.190:32797
I20260621 05:14:21.317178 21715 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.131.190:32797 every 8 connection(s)
I20260621 05:14:21.317517 21648 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782018616648578-3598-0/minicluster-data/master-0/data/info.pb
I20260621 05:14:21.318337 21716 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 05:14:21.320750 21716 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 65b67b67bb4e44dea10c37f4dd2af5ec: Bootstrap starting.
I20260621 05:14:21.321343 21716 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 65b67b67bb4e44dea10c37f4dd2af5ec: Neither blocks nor log segments found. Creating new log.
I20260621 05:14:21.321638 21716 log.cc:824] T 00000000000000000000000000000000 P 65b67b67bb4e44dea10c37f4dd2af5ec: Log is configured to *not* fsync() on all Append() calls
I20260621 05:14:21.322352 21716 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 65b67b67bb4e44dea10c37f4dd2af5ec: No bootstrap required, opened a new log
I20260621 05:14:21.324090 21716 raft_consensus.cc:348] T 00000000000000000000000000000000 P 65b67b67bb4e44dea10c37f4dd2af5ec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "65b67b67bb4e44dea10c37f4dd2af5ec" member_type: VOTER last_known_addr { host: "127.3.131.190" port: 32797 } }
I20260621 05:14:21.324242 21716 raft_consensus.cc:374] T 00000000000000000000000000000000 P 65b67b67bb4e44dea10c37f4dd2af5ec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 05:14:21.324266 21716 raft_consensus.cc:729] T 00000000000000000000000000000000 P 65b67b67bb4e44dea10c37f4dd2af5ec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 65b67b67bb4e44dea10c37f4dd2af5ec, State: Initialized, Role: FOLLOWER
I20260621 05:14:21.324429 21716 consensus_queue.cc:260] T 00000000000000000000000000000000 P 65b67b67bb4e44dea10c37f4dd2af5ec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "65b67b67bb4e44dea10c37f4dd2af5ec" member_type: VOTER last_known_addr { host: "127.3.131.190" port: 32797 } }
I20260621 05:14:21.324501 21716 raft_consensus.cc:388] T 00000000000000000000000000000000 P 65b67b67bb4e44dea10c37f4dd2af5ec [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260621 05:14:21.324541 21716 raft_consensus.cc:482] T 00000000000000000000000000000000 P 65b67b67bb4e44dea10c37f4dd2af5ec [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260621 05:14:21.324594 21716 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 65b67b67bb4e44dea10c37f4dd2af5ec [term 0 FOLLOWER]: Advancing to term 1
I20260621 05:14:21.325184 21716 raft_consensus.cc:504] T 00000000000000000000000000000000 P 65b67b67bb4e44dea10c37f4dd2af5ec [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "65b67b67bb4e44dea10c37f4dd2af5ec" member_type: VOTER last_known_addr { host: "127.3.131.190" port: 32797 } }
I20260621 05:14:21.325274 21716 leader_election.cc:302] T 00000000000000000000000000000000 P 65b67b67bb4e44dea10c37f4dd2af5ec [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 65b67b67bb4e44dea10c37f4dd2af5ec; no voters:
I20260621 05:14:21.325467 21716 leader_election.cc:288] T 00000000000000000000000000000000 P 65b67b67bb4e44dea10c37f4dd2af5ec [CANDIDATE]: Term 1 election: Requested vote from peers
I20260621 05:14:21.325563 21719 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 65b67b67bb4e44dea10c37f4dd2af5ec [term 1 FOLLOWER]: Leader election won for term 1
I20260621 05:14:21.325719 21719 raft_consensus.cc:686] T 00000000000000000000000000000000 P 65b67b67bb4e44dea10c37f4dd2af5ec [term 1 LEADER]: Becoming Leader. State: Replica: 65b67b67bb4e44dea10c37f4dd2af5ec, State: Running, Role: LEADER
I20260621 05:14:21.325814 21716 sys_catalog.cc:564] T 00000000000000000000000000000000 P 65b67b67bb4e44dea10c37f4dd2af5ec [sys.catalog]: configured and running, proceeding with master startup.
I20260621 05:14:21.325814 21719 consensus_queue.cc:237] T 00000000000000000000000000000000 P 65b67b67bb4e44dea10c37f4dd2af5ec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "65b67b67bb4e44dea10c37f4dd2af5ec" member_type: VOTER last_known_addr { host: "127.3.131.190" port: 32797 } }
I20260621 05:14:21.326471 21721 sys_catalog.cc:455] T 00000000000000000000000000000000 P 65b67b67bb4e44dea10c37f4dd2af5ec [sys.catalog]: SysCatalogTable state changed. Reason: New leader 65b67b67bb4e44dea10c37f4dd2af5ec. Latest consensus state: current_term: 1 leader_uuid: "65b67b67bb4e44dea10c37f4dd2af5ec" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "65b67b67bb4e44dea10c37f4dd2af5ec" member_type: VOTER last_known_addr { host: "127.3.131.190" port: 32797 } } }
I20260621 05:14:21.326582 21721 sys_catalog.cc:458] T 00000000000000000000000000000000 P 65b67b67bb4e44dea10c37f4dd2af5ec [sys.catalog]: This master's current role is: LEADER
I20260621 05:14:21.326740 3598 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWZOx2_/build/release/bin/kudu as pid 21648
I20260621 05:14:21.326757 21720 sys_catalog.cc:455] T 00000000000000000000000000000000 P 65b67b67bb4e44dea10c37f4dd2af5ec [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "65b67b67bb4e44dea10c37f4dd2af5ec" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "65b67b67bb4e44dea10c37f4dd2af5ec" member_type: VOTER last_known_addr { host: "127.3.131.190" port: 32797 } } }
I20260621 05:14:21.326814 21720 sys_catalog.cc:458] T 00000000000000000000000000000000 P 65b67b67bb4e44dea10c37f4dd2af5ec [sys.catalog]: This master's current role is: LEADER
I20260621 05:14:21.326843 3598 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782018616648578-3598-0/minicluster-data/master-0/wal/instance
I20260621 05:14:21.326900 21728 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260621 05:14:21.327071 3598 external_mini_cluster.cc:1428] Setting key 46d395fb632af0f3c8c91b7e6a617359
I20260621 05:14:21.327526 21728 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260621 05:14:21.329101 21728 catalog_manager.cc:1347] Generated new cluster ID: f4049b7e99c241a4895fdc88220cf8d4
I20260621 05:14:21.329159 21728 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Jun 21 05:14:21 dist-test-slave-vrq1 krb5kdc[21635](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782018861, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.131.190@KRBTEST.COM
I20260621 05:14:21.337631 21738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:14:21.327804 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40450 (local address 127.3.131.190:32797)
0621 05:14:21.328255 (+ 451us) server_negotiation.cc:206] Beginning negotiation
0621 05:14:21.328265 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0621 05:14:21.328304 (+ 39us) server_negotiation.cc:407] Connection header received
0621 05:14:21.328423 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:14:21.328430 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:14:21.328495 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:14:21.328639 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:14:21.329429 (+ 790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:14:21.330366 (+ 937us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:14:21.331135 (+ 769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:14:21.331674 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:14:21.331790 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:14:21.334213 (+ 2423us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:14:21.334218 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:14:21.334218 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:14:21.334230 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:14:21.335991 (+ 1761us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:14:21.336381 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:14:21.336385 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:14:21.336385 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:14:21.336494 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:14:21.336717 (+ 223us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:14:21.336718 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:14:21.336718 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:14:21.336983 (+ 265us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:14:21.337161 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0621 05:14:21.337245 (+ 84us) server_negotiation.cc:299] Negotiation successful
0621 05:14:21.337395 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":399,"thread_start_us":69,"threads_started":1}
I20260621 05:14:21.362553 21728 catalog_manager.cc:1370] Generated new certificate authority record
I20260621 05:14:21.363240 21728 catalog_manager.cc:1504] Loading token signing keys...
I20260621 05:14:21.369663 21728 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 65b67b67bb4e44dea10c37f4dd2af5ec: Generated new TSK 0
I20260621 05:14:21.369871 21728 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.3.131.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.131.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.131.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782018616648578-3598-0/krb5kdc/kudu_127.3.131.129.keytab.
Entry for principal kudu/127.3.131.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782018616648578-3598-0/krb5kdc/kudu_127.3.131.129.keytab.
I20260621 05:14:21.404270 3598 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWZOx2_/build/release/bin/kudu
/tmp/dist-test-taskWZOx2_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782018616648578-3598-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782018616648578-3598-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782018616648578-3598-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782018616648578-3598-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.131.129:0
--local_ip_for_outbound_sockets=127.3.131.129
--webserver_interface=127.3.131.129
--webserver_port=0
--tserver_master_addrs=127.3.131.190:32797
--keytab_file=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782018616648578-3598-0/krb5kdc/kudu_127.3.131.129.keytab
--principal=kudu/127.3.131.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.131.148:40107
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782018616648578-3598-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782018616648578-3598-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782018616648578-3598-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 05:14:21.491271 21742 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 05:14:21.491509 21742 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 05:14:21.491539 21742 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 05:14:21.492802 21742 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 05:14:21.492848 21742 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 05:14:21.492877 21742 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.131.129
I20260621 05:14:21.494516 21742 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.131.148:40107
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782018616648578-3598-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782018616648578-3598-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782018616648578-3598-0/krb5kdc/kudu_127.3.131.129.keytab
--principal=kudu/127.3.131.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.131.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782018616648578-3598-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.131.129
--webserver_port=0
--tserver_master_addrs=127.3.131.190:32797
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21742
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.131.129
--log_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782018616648578-3598-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 21 Jun 2026 04:16:16 UTC on ccycloud.quasar-xycaha.root.comops.site
build id 4457881
I20260621 05:14:21.494756 21742 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 05:14:21.494954 21742 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 05:14:21.500775 21747 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:14:21.501935 21742 server_base.cc:1034] running on GCE node
W20260621 05:14:21.502604 21748 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:14:21.502789 21750 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:14:21.503270 21742 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 05:14:21.503793 21742 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 05:14:21.504981 21742 hybrid_clock.cc:648] HybridClock initialized: now 1782018861504973 us; error 34 us; skew 500 ppm
Jun 21 05:14:21 dist-test-slave-vrq1 krb5kdc[21635](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782018861, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.131.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 05:14:21.508126 21742 init.cc:377] Logged in from keytab as kudu/127.3.131.129@KRBTEST.COM (short username kudu)
I20260621 05:14:21.508642 21742 webserver.cc:460] Webserver started at http://127.3.131.129:41663/ using document root <none> and password file <none>
I20260621 05:14:21.508879 21742 fs_manager.cc:359] Metadata directory not provided
I20260621 05:14:21.508925 21742 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 05:14:21.509021 21742 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 05:14:21.510111 21742 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782018616648578-3598-0/minicluster-data/ts-0/data/instance:
uuid: "574266cc91594bd5a2465fb6ffb6f661"
format_stamp: "Formatted at 2026-06-21 05:14:21 on dist-test-slave-vrq1"
server_key: "0da7d6f8a48d11dbcfcd259c970984e8"
server_key_iv: "863927e9eeb2cd4a1391e6d040f4d569"
server_key_version: "encryptionkey@0"
I20260621 05:14:21.510433 21742 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782018616648578-3598-0/minicluster-data/ts-0/wal/instance:
uuid: "574266cc91594bd5a2465fb6ffb6f661"
format_stamp: "Formatted at 2026-06-21 05:14:21 on dist-test-slave-vrq1"
server_key: "0da7d6f8a48d11dbcfcd259c970984e8"
server_key_iv: "863927e9eeb2cd4a1391e6d040f4d569"
server_key_version: "encryptionkey@0"
I20260621 05:14:21.511885 21742 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260621 05:14:21.512742 21757 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 05:14:21.512975 21742 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260621 05:14:21.513084 21742 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782018616648578-3598-0/minicluster-data/ts-0/data,/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782018616648578-3598-0/minicluster-data/ts-0/wal
uuid: "574266cc91594bd5a2465fb6ffb6f661"
format_stamp: "Formatted at 2026-06-21 05:14:21 on dist-test-slave-vrq1"
server_key: "0da7d6f8a48d11dbcfcd259c970984e8"
server_key_iv: "863927e9eeb2cd4a1391e6d040f4d569"
server_key_version: "encryptionkey@0"
I20260621 05:14:21.513171 21742 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782018616648578-3598-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782018616648578-3598-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782018616648578-3598-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 05:14:21.531590 21742 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 05:14:21.534039 21742 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 05:14:21.534215 21742 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 05:14:21.534483 21742 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 05:14:21.534801 21742 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 05:14:21.534852 21742 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 05:14:21.534894 21742 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 05:14:21.534932 21742 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 05:14:21.541013 21742 rpc_server.cc:307] RPC server started. Bound to: 127.3.131.129:41197
I20260621 05:14:21.541057 21870 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.131.129:41197 every 8 connection(s)
I20260621 05:14:21.541381 21742 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782018616648578-3598-0/minicluster-data/ts-0/data/info.pb
Jun 21 05:14:21 dist-test-slave-vrq1 krb5kdc[21635](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782018861, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.131.129@KRBTEST.COM for kudu/127.3.131.190@KRBTEST.COM
I20260621 05:14:21.550159 3598 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWZOx2_/build/release/bin/kudu as pid 21742
I20260621 05:14:21.550292 3598 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782018616648578-3598-0/minicluster-data/ts-0/wal/instance
I20260621 05:14:21.550575 3598 external_mini_cluster.cc:1428] Setting key 278dfcd28ea73bf1e5e70fb6bd23aec2
I20260621 05:14:21.551198 21738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:14:21.542085 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.131.129:40865 (local address 127.3.131.190:32797)
0621 05:14:21.542131 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0621 05:14:21.542133 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:14:21.542335 (+ 202us) server_negotiation.cc:407] Connection header received
0621 05:14:21.542827 (+ 492us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:14:21.542828 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:14:21.542838 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:14:21.542889 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:14:21.543843 (+ 954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:14:21.544357 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:14:21.545393 (+ 1036us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:14:21.545582 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:14:21.545633 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:14:21.548005 (+ 2372us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:14:21.548007 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:14:21.548007 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:14:21.548014 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:14:21.549452 (+ 1438us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:14:21.549929 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:14:21.549930 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:14:21.549930 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:14:21.549987 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:14:21.550367 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:14:21.550368 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:14:21.550369 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:14:21.550500 (+ 131us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:14:21.550562 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0621 05:14:21.550944 (+ 382us) server_negotiation.cc:299] Negotiation successful
0621 05:14:21.551056 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260621 05:14:21.551432 21873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:14:21.542078 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.131.190:32797 (local address 127.3.131.129:40865)
0621 05:14:21.542219 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0621 05:14:21.542254 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0621 05:14:21.542742 (+ 488us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:14:21.542944 (+ 202us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:14:21.542948 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:14:21.543542 (+ 594us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:14:21.543758 (+ 216us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:14:21.543763 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:14:21.544445 (+ 682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:14:21.544446 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:14:21.545317 (+ 871us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:14:21.545320 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:14:21.545680 (+ 360us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:14:21.545682 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:14:21.545805 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:14:21.546414 (+ 609us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 05:14:21.546423 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:14:21.547900 (+ 1477us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 05:14:21.549543 (+ 1643us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:14:21.549550 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:14:21.549553 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:14:21.549804 (+ 251us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:14:21.550070 (+ 266us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:14:21.550071 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:14:21.550072 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:14:21.550241 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:14:21.550575 (+ 334us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 05:14:21.550582 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 05:14:21.550764 (+ 182us) client_negotiation.cc:769] Sending connection context
0621 05:14:21.550926 (+ 162us) client_negotiation.cc:240] Negotiation successful
0621 05:14:21.551113 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":87,"threads_started":1}
I20260621 05:14:21.552047 21871 heartbeater.cc:347] Connected to a master server at 127.3.131.190:32797
I20260621 05:14:21.552163 21871 heartbeater.cc:464] Registering TS with master...
I20260621 05:14:21.552359 21871 heartbeater.cc:511] Master 127.3.131.190:32797 requested a full tablet report, sending...
I20260621 05:14:21.552979 21680 ts_manager.cc:194] Registered new tserver with Master: 574266cc91594bd5a2465fb6ffb6f661 (127.3.131.129:41197)
I20260621 05:14:21.553601 21680 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.131.129@KRBTEST.COM'} at 127.3.131.129:40865
WARNING: no policy specified for kudu/127.3.131.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.131.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.131.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782018616648578-3598-0/krb5kdc/kudu_127.3.131.130.keytab.
Entry for principal kudu/127.3.131.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782018616648578-3598-0/krb5kdc/kudu_127.3.131.130.keytab.
I20260621 05:14:21.578629 3598 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWZOx2_/build/release/bin/kudu
/tmp/dist-test-taskWZOx2_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782018616648578-3598-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782018616648578-3598-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782018616648578-3598-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782018616648578-3598-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.131.130:0
--local_ip_for_outbound_sockets=127.3.131.130
--webserver_interface=127.3.131.130
--webserver_port=0
--tserver_master_addrs=127.3.131.190:32797
--keytab_file=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782018616648578-3598-0/krb5kdc/kudu_127.3.131.130.keytab
--principal=kudu/127.3.131.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.131.148:40107
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782018616648578-3598-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782018616648578-3598-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782018616648578-3598-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 05:14:21.667074 21876 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 05:14:21.667397 21876 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 05:14:21.667438 21876 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 05:14:21.668679 21876 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 05:14:21.668763 21876 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 05:14:21.668820 21876 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.131.130
I20260621 05:14:21.670329 21876 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.131.148:40107
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782018616648578-3598-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782018616648578-3598-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782018616648578-3598-0/krb5kdc/kudu_127.3.131.130.keytab
--principal=kudu/127.3.131.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.131.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782018616648578-3598-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.131.130
--webserver_port=0
--tserver_master_addrs=127.3.131.190:32797
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21876
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.131.130
--log_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782018616648578-3598-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 21 Jun 2026 04:16:16 UTC on ccycloud.quasar-xycaha.root.comops.site
build id 4457881
I20260621 05:14:21.670593 21876 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 05:14:21.670830 21876 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 05:14:21.676266 21882 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:14:21.676261 21881 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:14:21.676388 21884 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:14:21.676488 21876 server_base.cc:1034] running on GCE node
I20260621 05:14:21.676718 21876 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 05:14:21.676968 21876 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 05:14:21.678150 21876 hybrid_clock.cc:648] HybridClock initialized: now 1782018861678135 us; error 31 us; skew 500 ppm
Jun 21 05:14:21 dist-test-slave-vrq1 krb5kdc[21635](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782018861, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.131.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 05:14:21.680805 21876 init.cc:377] Logged in from keytab as kudu/127.3.131.130@KRBTEST.COM (short username kudu)
I20260621 05:14:21.681303 21876 webserver.cc:460] Webserver started at http://127.3.131.130:45757/ using document root <none> and password file <none>
I20260621 05:14:21.681495 21876 fs_manager.cc:359] Metadata directory not provided
I20260621 05:14:21.681537 21876 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 05:14:21.681641 21876 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 05:14:21.682718 21876 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782018616648578-3598-0/minicluster-data/ts-1/data/instance:
uuid: "e62ab0b7280143c799337c11235d51cb"
format_stamp: "Formatted at 2026-06-21 05:14:21 on dist-test-slave-vrq1"
server_key: "749decd24ebb9c8227045a9b7cfea1dc"
server_key_iv: "b059607ea75f52d77ccab6df05fb41c8"
server_key_version: "encryptionkey@0"
I20260621 05:14:21.683036 21876 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782018616648578-3598-0/minicluster-data/ts-1/wal/instance:
uuid: "e62ab0b7280143c799337c11235d51cb"
format_stamp: "Formatted at 2026-06-21 05:14:21 on dist-test-slave-vrq1"
server_key: "749decd24ebb9c8227045a9b7cfea1dc"
server_key_iv: "b059607ea75f52d77ccab6df05fb41c8"
server_key_version: "encryptionkey@0"
I20260621 05:14:21.684538 21876 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260621 05:14:21.685390 21891 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 05:14:21.685609 21876 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260621 05:14:21.685757 21876 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782018616648578-3598-0/minicluster-data/ts-1/data,/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782018616648578-3598-0/minicluster-data/ts-1/wal
uuid: "e62ab0b7280143c799337c11235d51cb"
format_stamp: "Formatted at 2026-06-21 05:14:21 on dist-test-slave-vrq1"
server_key: "749decd24ebb9c8227045a9b7cfea1dc"
server_key_iv: "b059607ea75f52d77ccab6df05fb41c8"
server_key_version: "encryptionkey@0"
I20260621 05:14:21.685833 21876 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782018616648578-3598-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782018616648578-3598-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782018616648578-3598-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 05:14:21.751765 21876 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 05:14:21.754122 21876 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 05:14:21.754240 21876 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 05:14:21.754516 21876 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 05:14:21.754838 21876 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 05:14:21.754873 21876 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 05:14:21.754895 21876 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 05:14:21.754933 21876 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 05:14:21.760834 21876 rpc_server.cc:307] RPC server started. Bound to: 127.3.131.130:36511
I20260621 05:14:21.760881 22004 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.131.130:36511 every 8 connection(s)
I20260621 05:14:21.761190 21876 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782018616648578-3598-0/minicluster-data/ts-1/data/info.pb
I20260621 05:14:21.765571 3598 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWZOx2_/build/release/bin/kudu as pid 21876
I20260621 05:14:21.765690 3598 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782018616648578-3598-0/minicluster-data/ts-1/wal/instance
I20260621 05:14:21.765935 3598 external_mini_cluster.cc:1428] Setting key 5eb7c6f86491b6a80d2e70b156d48bf6
Jun 21 05:14:21 dist-test-slave-vrq1 krb5kdc[21635](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782018861, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.131.130@KRBTEST.COM for kudu/127.3.131.190@KRBTEST.COM
I20260621 05:14:21.771832 21738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:14:21.761867 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.131.130:35255 (local address 127.3.131.190:32797)
0621 05:14:21.761917 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0621 05:14:21.761919 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:14:21.762088 (+ 169us) server_negotiation.cc:407] Connection header received
0621 05:14:21.762586 (+ 498us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:14:21.762587 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:14:21.762599 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:14:21.762668 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:14:21.763674 (+ 1006us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:14:21.764184 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:14:21.765161 (+ 977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:14:21.765392 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:14:21.765454 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:14:21.768355 (+ 2901us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:14:21.768357 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:14:21.768358 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:14:21.768366 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:14:21.769990 (+ 1624us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:14:21.770588 (+ 598us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:14:21.770589 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:14:21.770589 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:14:21.770649 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:14:21.771033 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:14:21.771034 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:14:21.771035 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:14:21.771162 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:14:21.771281 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0621 05:14:21.771660 (+ 379us) server_negotiation.cc:299] Negotiation successful
0621 05:14:21.771764 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260621 05:14:21.772143 22007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:14:21.761809 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.131.190:32797 (local address 127.3.131.130:35255)
0621 05:14:21.761965 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0621 05:14:21.762007 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0621 05:14:21.762474 (+ 467us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:14:21.762741 (+ 267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:14:21.762744 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:14:21.763380 (+ 636us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:14:21.763578 (+ 198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:14:21.763583 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:14:21.764264 (+ 681us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:14:21.764265 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:14:21.765066 (+ 801us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:14:21.765068 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:14:21.765496 (+ 428us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:14:21.765498 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:14:21.765698 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:14:21.766401 (+ 703us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 05:14:21.766414 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:14:21.768228 (+ 1814us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 05:14:21.770090 (+ 1862us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:14:21.770098 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:14:21.770101 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:14:21.770469 (+ 368us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:14:21.770729 (+ 260us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:14:21.770730 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:14:21.770730 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:14:21.770900 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:14:21.771285 (+ 385us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 05:14:21.771293 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 05:14:21.771493 (+ 200us) client_negotiation.cc:769] Sending connection context
0621 05:14:21.771622 (+ 129us) client_negotiation.cc:240] Negotiation successful
0621 05:14:21.771840 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":82,"threads_started":1}
I20260621 05:14:21.772915 22005 heartbeater.cc:347] Connected to a master server at 127.3.131.190:32797
I20260621 05:14:21.773034 22005 heartbeater.cc:464] Registering TS with master...
I20260621 05:14:21.773262 22005 heartbeater.cc:511] Master 127.3.131.190:32797 requested a full tablet report, sending...
I20260621 05:14:21.773758 21680 ts_manager.cc:194] Registered new tserver with Master: e62ab0b7280143c799337c11235d51cb (127.3.131.130:36511)
I20260621 05:14:21.774148 21680 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.131.130@KRBTEST.COM'} at 127.3.131.130:35255
WARNING: no policy specified for kudu/127.3.131.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.131.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.131.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782018616648578-3598-0/krb5kdc/kudu_127.3.131.131.keytab.
Entry for principal kudu/127.3.131.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782018616648578-3598-0/krb5kdc/kudu_127.3.131.131.keytab.
I20260621 05:14:21.795753 3598 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWZOx2_/build/release/bin/kudu
/tmp/dist-test-taskWZOx2_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782018616648578-3598-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782018616648578-3598-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782018616648578-3598-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782018616648578-3598-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.131.131:0
--local_ip_for_outbound_sockets=127.3.131.131
--webserver_interface=127.3.131.131
--webserver_port=0
--tserver_master_addrs=127.3.131.190:32797
--keytab_file=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782018616648578-3598-0/krb5kdc/kudu_127.3.131.131.keytab
--principal=kudu/127.3.131.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.131.148:40107
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782018616648578-3598-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782018616648578-3598-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782018616648578-3598-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 05:14:21.881098 22010 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 05:14:21.881386 22010 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 05:14:21.881430 22010 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 05:14:21.882719 22010 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 05:14:21.882791 22010 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 05:14:21.882839 22010 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.131.131
I20260621 05:14:21.884542 22010 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.131.148:40107
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782018616648578-3598-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782018616648578-3598-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782018616648578-3598-0/krb5kdc/kudu_127.3.131.131.keytab
--principal=kudu/127.3.131.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.131.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782018616648578-3598-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.131.131
--webserver_port=0
--tserver_master_addrs=127.3.131.190:32797
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22010
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.131.131
--log_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782018616648578-3598-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 21 Jun 2026 04:16:16 UTC on ccycloud.quasar-xycaha.root.comops.site
build id 4457881
I20260621 05:14:21.884855 22010 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 05:14:21.885097 22010 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 05:14:21.890470 22015 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:14:21.890712 22016 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:14:21.890599 22018 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:14:21.890775 22010 server_base.cc:1034] running on GCE node
I20260621 05:14:21.891088 22010 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 05:14:21.891438 22010 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 05:14:21.892657 22010 hybrid_clock.cc:648] HybridClock initialized: now 1782018861892610 us; error 66 us; skew 500 ppm
Jun 21 05:14:21 dist-test-slave-vrq1 krb5kdc[21635](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782018861, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.131.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 05:14:21.895413 22010 init.cc:377] Logged in from keytab as kudu/127.3.131.131@KRBTEST.COM (short username kudu)
I20260621 05:14:21.895843 22010 webserver.cc:460] Webserver started at http://127.3.131.131:34623/ using document root <none> and password file <none>
I20260621 05:14:21.896046 22010 fs_manager.cc:359] Metadata directory not provided
I20260621 05:14:21.896088 22010 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 05:14:21.896232 22010 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 05:14:21.897297 22010 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782018616648578-3598-0/minicluster-data/ts-2/data/instance:
uuid: "65b483c0d81042a89adcd922412a4e10"
format_stamp: "Formatted at 2026-06-21 05:14:21 on dist-test-slave-vrq1"
server_key: "3be2f176efa210e9b09b1cd00cd83e21"
server_key_iv: "2f19d38794ff8655c9319ef6a6389c4d"
server_key_version: "encryptionkey@0"
I20260621 05:14:21.897624 22010 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782018616648578-3598-0/minicluster-data/ts-2/wal/instance:
uuid: "65b483c0d81042a89adcd922412a4e10"
format_stamp: "Formatted at 2026-06-21 05:14:21 on dist-test-slave-vrq1"
server_key: "3be2f176efa210e9b09b1cd00cd83e21"
server_key_iv: "2f19d38794ff8655c9319ef6a6389c4d"
server_key_version: "encryptionkey@0"
I20260621 05:14:21.899041 22010 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260621 05:14:21.900036 22025 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 05:14:21.900342 22010 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260621 05:14:21.900449 22010 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782018616648578-3598-0/minicluster-data/ts-2/data,/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782018616648578-3598-0/minicluster-data/ts-2/wal
uuid: "65b483c0d81042a89adcd922412a4e10"
format_stamp: "Formatted at 2026-06-21 05:14:21 on dist-test-slave-vrq1"
server_key: "3be2f176efa210e9b09b1cd00cd83e21"
server_key_iv: "2f19d38794ff8655c9319ef6a6389c4d"
server_key_version: "encryptionkey@0"
I20260621 05:14:21.900507 22010 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782018616648578-3598-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782018616648578-3598-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782018616648578-3598-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 05:14:21.913046 22010 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 05:14:21.915105 22010 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 05:14:21.915318 22010 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 05:14:21.915521 22010 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 05:14:21.915932 22010 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 05:14:21.915977 22010 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 05:14:21.916020 22010 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 05:14:21.916046 22010 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 05:14:21.921751 22010 rpc_server.cc:307] RPC server started. Bound to: 127.3.131.131:39477
I20260621 05:14:21.921838 22138 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.131.131:39477 every 8 connection(s)
I20260621 05:14:21.922155 22010 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782018616648578-3598-0/minicluster-data/ts-2/data/info.pb
Jun 21 05:14:21 dist-test-slave-vrq1 krb5kdc[21635](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782018861, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.131.131@KRBTEST.COM for kudu/127.3.131.190@KRBTEST.COM
I20260621 05:14:21.931699 3598 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWZOx2_/build/release/bin/kudu as pid 22010
I20260621 05:14:21.931823 3598 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782018616648578-3598-0/minicluster-data/ts-2/wal/instance
I20260621 05:14:21.932053 3598 external_mini_cluster.cc:1428] Setting key 11c8db5cc5883ac39ab136fa26f2140b
I20260621 05:14:21.932411 21738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:14:21.922886 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.131.131:37093 (local address 127.3.131.190:32797)
0621 05:14:21.922952 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0621 05:14:21.922954 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:14:21.923257 (+ 303us) server_negotiation.cc:407] Connection header received
0621 05:14:21.923749 (+ 492us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:14:21.923751 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:14:21.923763 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:14:21.923828 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:14:21.924838 (+ 1010us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:14:21.925426 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:14:21.926666 (+ 1240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:14:21.926936 (+ 270us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:14:21.926998 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:14:21.929425 (+ 2427us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:14:21.929426 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:14:21.929427 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:14:21.929434 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:14:21.930867 (+ 1433us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:14:21.931348 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:14:21.931349 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:14:21.931349 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:14:21.931415 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:14:21.931694 (+ 279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:14:21.931695 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:14:21.931695 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:14:21.931835 (+ 140us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:14:21.931900 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0621 05:14:21.932232 (+ 332us) server_negotiation.cc:299] Negotiation successful
0621 05:14:21.932315 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260621 05:14:21.932632 22141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:14:21.922854 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.131.190:32797 (local address 127.3.131.131:37093)
0621 05:14:21.923042 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0621 05:14:21.923085 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0621 05:14:21.923634 (+ 549us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:14:21.923898 (+ 264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:14:21.923902 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:14:21.924486 (+ 584us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:14:21.924685 (+ 199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:14:21.924689 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:14:21.925532 (+ 843us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:14:21.925533 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:14:21.926568 (+ 1035us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:14:21.926571 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:14:21.927007 (+ 436us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:14:21.927008 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:14:21.927160 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:14:21.927777 (+ 617us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 05:14:21.927787 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:14:21.929345 (+ 1558us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 05:14:21.930960 (+ 1615us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:14:21.930966 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:14:21.930968 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:14:21.931276 (+ 308us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:14:21.931482 (+ 206us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:14:21.931483 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:14:21.931483 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:14:21.931605 (+ 122us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:14:21.931928 (+ 323us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 05:14:21.931933 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 05:14:21.932062 (+ 129us) client_negotiation.cc:769] Sending connection context
0621 05:14:21.932162 (+ 100us) client_negotiation.cc:240] Negotiation successful
0621 05:14:21.932365 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":115,"threads_started":1}
I20260621 05:14:21.933151 22139 heartbeater.cc:347] Connected to a master server at 127.3.131.190:32797
I20260621 05:14:21.933255 22139 heartbeater.cc:464] Registering TS with master...
I20260621 05:14:21.933441 22139 heartbeater.cc:511] Master 127.3.131.190:32797 requested a full tablet report, sending...
I20260621 05:14:21.933772 21679 ts_manager.cc:194] Registered new tserver with Master: 65b483c0d81042a89adcd922412a4e10 (127.3.131.131:39477)
I20260621 05:14:21.934168 21679 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.131.131@KRBTEST.COM'} at 127.3.131.131:37093
I20260621 05:14:21.944574 3598 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260621 05:14:21.952643 21738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:14:21.946068 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46450 (local address 127.3.131.190:32797)
0621 05:14:21.946141 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0621 05:14:21.946143 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:14:21.946237 (+ 94us) server_negotiation.cc:407] Connection header received
0621 05:14:21.946283 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:14:21.946283 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:14:21.946294 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:14:21.946346 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:14:21.946875 (+ 529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:14:21.947479 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:14:21.948213 (+ 734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:14:21.948400 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:14:21.948458 (+ 58us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:14:21.949997 (+ 1539us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:14:21.949998 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:14:21.949998 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:14:21.950005 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:14:21.951472 (+ 1467us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:14:21.951935 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:14:21.951936 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:14:21.951937 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:14:21.951990 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:14:21.952179 (+ 189us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:14:21.952179 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:14:21.952179 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:14:21.952319 (+ 140us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:14:21.952403 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0621 05:14:21.952530 (+ 127us) server_negotiation.cc:299] Negotiation successful
0621 05:14:21.952578 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260621 05:14:21.954432 21679 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46450:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260621 05:14:21.956144 21679 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260621 05:14:21.962797 22152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:14:21.958779 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49976 (local address 127.3.131.130:36511)
0621 05:14:21.958992 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0621 05:14:21.958998 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 05:14:21.959164 (+ 166us) server_negotiation.cc:407] Connection header received
0621 05:14:21.959263 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:14:21.959270 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:14:21.959291 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:14:21.959386 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 05:14:21.959588 (+ 202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:14:21.960390 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:14:21.961757 (+ 1367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:14:21.962389 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:14:21.962424 (+ 35us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:14:21.962462 (+ 38us) server_negotiation.cc:1035] Waiting for connection context
0621 05:14:21.962670 (+ 208us) server_negotiation.cc:299] Negotiation successful
0621 05:14:21.962710 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":181,"thread_start_us":98,"threads_started":1}
I20260621 05:14:21.962790 22153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:14:21.958615 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.131.130:36511 (local address 127.0.0.1:49976)
0621 05:14:21.959106 (+ 491us) negotiation.cc:107] Waiting for socket to connect
0621 05:14:21.959115 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0621 05:14:21.959193 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:14:21.959399 (+ 206us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:14:21.959403 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:14:21.959406 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 05:14:21.959489 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:14:21.959491 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:14:21.960487 (+ 996us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:14:21.960488 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:14:21.961647 (+ 1159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:14:21.961649 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:14:21.962510 (+ 861us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:14:21.962511 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:14:21.962566 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:14:21.962575 (+ 9us) client_negotiation.cc:769] Sending connection context
0621 05:14:21.962652 (+ 77us) client_negotiation.cc:240] Negotiation successful
0621 05:14:21.962710 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":472,"mutex_wait_us":3,"thread_start_us":73,"threads_started":1}
I20260621 05:14:21.962993 22150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:14:21.958817 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.131.129:41197 (local address 127.0.0.1:59440)
0621 05:14:21.959154 (+ 337us) negotiation.cc:107] Waiting for socket to connect
0621 05:14:21.959162 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0621 05:14:21.959267 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:14:21.959525 (+ 258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:14:21.959527 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:14:21.959529 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 05:14:21.959582 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:14:21.959584 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:14:21.960971 (+ 1387us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:14:21.960973 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:14:21.961857 (+ 884us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:14:21.961859 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:14:21.962735 (+ 876us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:14:21.962736 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:14:21.962799 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:14:21.962804 (+ 5us) client_negotiation.cc:769] Sending connection context
0621 05:14:21.962844 (+ 40us) client_negotiation.cc:240] Negotiation successful
0621 05:14:21.962890 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":328,"mutex_wait_us":1,"thread_start_us":66,"threads_started":1}
I20260621 05:14:21.963346 22155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:14:21.958942 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59440 (local address 127.3.131.129:41197)
0621 05:14:21.959096 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0621 05:14:21.959099 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0621 05:14:21.959268 (+ 169us) server_negotiation.cc:407] Connection header received
0621 05:14:21.959345 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:14:21.959348 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:14:21.959363 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:14:21.959659 (+ 296us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 05:14:21.959771 (+ 112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:14:21.960839 (+ 1068us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:14:21.961943 (+ 1104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:14:21.962621 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:14:21.963098 (+ 477us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:14:21.963168 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0621 05:14:21.963223 (+ 55us) server_negotiation.cc:299] Negotiation successful
0621 05:14:21.963269 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":124,"thread_start_us":75,"threads_started":1}
I20260621 05:14:21.963663 22151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:14:21.958615 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.131.131:39477 (local address 127.0.0.1:42364)
0621 05:14:21.958948 (+ 333us) negotiation.cc:107] Waiting for socket to connect
0621 05:14:21.958992 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0621 05:14:21.959057 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:14:21.959552 (+ 495us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:14:21.959553 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:14:21.959554 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 05:14:21.959584 (+ 30us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:14:21.959584 (+ 0us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:14:21.960951 (+ 1367us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:14:21.960952 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:14:21.962026 (+ 1074us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:14:21.962028 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:14:21.963428 (+ 1400us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:14:21.963430 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:14:21.963491 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:14:21.963495 (+ 4us) client_negotiation.cc:769] Sending connection context
0621 05:14:21.963559 (+ 64us) client_negotiation.cc:240] Negotiation successful
0621 05:14:21.963596 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":302,"thread_start_us":65,"threads_started":1}
I20260621 05:14:21.963855 21939 tablet_service.cc:1467] Processing CreateTablet for tablet eb209814120a4e97a4fe05c18cd6d763 (DEFAULT_TABLE table=test-table [id=6049ac0a904d40dfa96e5678016c7c75]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 05:14:21.963979 21805 tablet_service.cc:1467] Processing CreateTablet for tablet eb209814120a4e97a4fe05c18cd6d763 (DEFAULT_TABLE table=test-table [id=6049ac0a904d40dfa96e5678016c7c75]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 05:14:21.964221 21939 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eb209814120a4e97a4fe05c18cd6d763. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 05:14:21.964267 21805 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eb209814120a4e97a4fe05c18cd6d763. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 05:14:21.966620 22156 tablet_bootstrap.cc:492] T eb209814120a4e97a4fe05c18cd6d763 P e62ab0b7280143c799337c11235d51cb: Bootstrap starting.
I20260621 05:14:21.966952 22157 tablet_bootstrap.cc:492] T eb209814120a4e97a4fe05c18cd6d763 P 574266cc91594bd5a2465fb6ffb6f661: Bootstrap starting.
I20260621 05:14:21.967095 22154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:14:21.958866 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42364 (local address 127.3.131.131:39477)
0621 05:14:21.959395 (+ 529us) server_negotiation.cc:206] Beginning negotiation
0621 05:14:21.959399 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 05:14:21.959407 (+ 8us) server_negotiation.cc:407] Connection header received
0621 05:14:21.959444 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:14:21.959446 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:14:21.959460 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:14:21.959655 (+ 195us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 05:14:21.959729 (+ 74us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:14:21.960839 (+ 1110us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:14:21.962104 (+ 1265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:14:21.962805 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:14:21.963407 (+ 602us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:14:21.963478 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0621 05:14:21.963580 (+ 102us) server_negotiation.cc:299] Negotiation successful
0621 05:14:21.967008 (+ 3428us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":490,"thread_start_us":92,"threads_started":1}
I20260621 05:14:21.967355 22156 tablet_bootstrap.cc:654] T eb209814120a4e97a4fe05c18cd6d763 P e62ab0b7280143c799337c11235d51cb: Neither blocks nor log segments found. Creating new log.
I20260621 05:14:21.967597 22157 tablet_bootstrap.cc:654] T eb209814120a4e97a4fe05c18cd6d763 P 574266cc91594bd5a2465fb6ffb6f661: Neither blocks nor log segments found. Creating new log.
I20260621 05:14:21.967926 22156 log.cc:824] T eb209814120a4e97a4fe05c18cd6d763 P e62ab0b7280143c799337c11235d51cb: Log is configured to *not* fsync() on all Append() calls
I20260621 05:14:21.967842 22073 tablet_service.cc:1467] Processing CreateTablet for tablet eb209814120a4e97a4fe05c18cd6d763 (DEFAULT_TABLE table=test-table [id=6049ac0a904d40dfa96e5678016c7c75]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 05:14:21.968007 22157 log.cc:824] T eb209814120a4e97a4fe05c18cd6d763 P 574266cc91594bd5a2465fb6ffb6f661: Log is configured to *not* fsync() on all Append() calls
I20260621 05:14:21.968170 22073 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eb209814120a4e97a4fe05c18cd6d763. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 05:14:21.968690 22157 tablet_bootstrap.cc:492] T eb209814120a4e97a4fe05c18cd6d763 P 574266cc91594bd5a2465fb6ffb6f661: No bootstrap required, opened a new log
I20260621 05:14:21.968689 22156 tablet_bootstrap.cc:492] T eb209814120a4e97a4fe05c18cd6d763 P e62ab0b7280143c799337c11235d51cb: No bootstrap required, opened a new log
I20260621 05:14:21.968755 22156 ts_tablet_manager.cc:1397] T eb209814120a4e97a4fe05c18cd6d763 P e62ab0b7280143c799337c11235d51cb: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260621 05:14:21.968755 22157 ts_tablet_manager.cc:1397] T eb209814120a4e97a4fe05c18cd6d763 P 574266cc91594bd5a2465fb6ffb6f661: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260621 05:14:21.970752 22160 tablet_bootstrap.cc:492] T eb209814120a4e97a4fe05c18cd6d763 P 65b483c0d81042a89adcd922412a4e10: Bootstrap starting.
I20260621 05:14:21.970711 22156 raft_consensus.cc:348] T eb209814120a4e97a4fe05c18cd6d763 P e62ab0b7280143c799337c11235d51cb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "65b483c0d81042a89adcd922412a4e10" member_type: VOTER last_known_addr { host: "127.3.131.131" port: 39477 } } peers { permanent_uuid: "e62ab0b7280143c799337c11235d51cb" member_type: VOTER last_known_addr { host: "127.3.131.130" port: 36511 } } peers { permanent_uuid: "574266cc91594bd5a2465fb6ffb6f661" member_type: VOTER last_known_addr { host: "127.3.131.129" port: 41197 } }
I20260621 05:14:21.970701 22157 raft_consensus.cc:348] T eb209814120a4e97a4fe05c18cd6d763 P 574266cc91594bd5a2465fb6ffb6f661 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "65b483c0d81042a89adcd922412a4e10" member_type: VOTER last_known_addr { host: "127.3.131.131" port: 39477 } } peers { permanent_uuid: "e62ab0b7280143c799337c11235d51cb" member_type: VOTER last_known_addr { host: "127.3.131.130" port: 36511 } } peers { permanent_uuid: "574266cc91594bd5a2465fb6ffb6f661" member_type: VOTER last_known_addr { host: "127.3.131.129" port: 41197 } }
I20260621 05:14:21.970878 22157 raft_consensus.cc:374] T eb209814120a4e97a4fe05c18cd6d763 P 574266cc91594bd5a2465fb6ffb6f661 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 05:14:21.970878 22156 raft_consensus.cc:374] T eb209814120a4e97a4fe05c18cd6d763 P e62ab0b7280143c799337c11235d51cb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 05:14:21.970913 22156 raft_consensus.cc:729] T eb209814120a4e97a4fe05c18cd6d763 P e62ab0b7280143c799337c11235d51cb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e62ab0b7280143c799337c11235d51cb, State: Initialized, Role: FOLLOWER
I20260621 05:14:21.970913 22157 raft_consensus.cc:729] T eb209814120a4e97a4fe05c18cd6d763 P 574266cc91594bd5a2465fb6ffb6f661 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 574266cc91594bd5a2465fb6ffb6f661, State: Initialized, Role: FOLLOWER
I20260621 05:14:21.971028 22156 consensus_queue.cc:260] T eb209814120a4e97a4fe05c18cd6d763 P e62ab0b7280143c799337c11235d51cb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "65b483c0d81042a89adcd922412a4e10" member_type: VOTER last_known_addr { host: "127.3.131.131" port: 39477 } } peers { permanent_uuid: "e62ab0b7280143c799337c11235d51cb" member_type: VOTER last_known_addr { host: "127.3.131.130" port: 36511 } } peers { permanent_uuid: "574266cc91594bd5a2465fb6ffb6f661" member_type: VOTER last_known_addr { host: "127.3.131.129" port: 41197 } }
I20260621 05:14:21.971040 22157 consensus_queue.cc:260] T eb209814120a4e97a4fe05c18cd6d763 P 574266cc91594bd5a2465fb6ffb6f661 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "65b483c0d81042a89adcd922412a4e10" member_type: VOTER last_known_addr { host: "127.3.131.131" port: 39477 } } peers { permanent_uuid: "e62ab0b7280143c799337c11235d51cb" member_type: VOTER last_known_addr { host: "127.3.131.130" port: 36511 } } peers { permanent_uuid: "574266cc91594bd5a2465fb6ffb6f661" member_type: VOTER last_known_addr { host: "127.3.131.129" port: 41197 } }
I20260621 05:14:21.971338 22156 ts_tablet_manager.cc:1428] T eb209814120a4e97a4fe05c18cd6d763 P e62ab0b7280143c799337c11235d51cb: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260621 05:14:21.971338 22157 ts_tablet_manager.cc:1428] T eb209814120a4e97a4fe05c18cd6d763 P 574266cc91594bd5a2465fb6ffb6f661: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260621 05:14:21.971464 22005 heartbeater.cc:503] Master 127.3.131.190:32797 was elected leader, sending a full tablet report...
I20260621 05:14:21.971479 22160 tablet_bootstrap.cc:654] T eb209814120a4e97a4fe05c18cd6d763 P 65b483c0d81042a89adcd922412a4e10: Neither blocks nor log segments found. Creating new log.
I20260621 05:14:21.971784 21871 heartbeater.cc:503] Master 127.3.131.190:32797 was elected leader, sending a full tablet report...
I20260621 05:14:21.972420 22160 log.cc:824] T eb209814120a4e97a4fe05c18cd6d763 P 65b483c0d81042a89adcd922412a4e10: Log is configured to *not* fsync() on all Append() calls
I20260621 05:14:21.973163 22160 tablet_bootstrap.cc:492] T eb209814120a4e97a4fe05c18cd6d763 P 65b483c0d81042a89adcd922412a4e10: No bootstrap required, opened a new log
I20260621 05:14:21.973223 22160 ts_tablet_manager.cc:1397] T eb209814120a4e97a4fe05c18cd6d763 P 65b483c0d81042a89adcd922412a4e10: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260621 05:14:21.974838 22160 raft_consensus.cc:348] T eb209814120a4e97a4fe05c18cd6d763 P 65b483c0d81042a89adcd922412a4e10 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "65b483c0d81042a89adcd922412a4e10" member_type: VOTER last_known_addr { host: "127.3.131.131" port: 39477 } } peers { permanent_uuid: "e62ab0b7280143c799337c11235d51cb" member_type: VOTER last_known_addr { host: "127.3.131.130" port: 36511 } } peers { permanent_uuid: "574266cc91594bd5a2465fb6ffb6f661" member_type: VOTER last_known_addr { host: "127.3.131.129" port: 41197 } }
I20260621 05:14:21.975060 22160 raft_consensus.cc:374] T eb209814120a4e97a4fe05c18cd6d763 P 65b483c0d81042a89adcd922412a4e10 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 05:14:21.975087 22160 raft_consensus.cc:729] T eb209814120a4e97a4fe05c18cd6d763 P 65b483c0d81042a89adcd922412a4e10 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 65b483c0d81042a89adcd922412a4e10, State: Initialized, Role: FOLLOWER
I20260621 05:14:21.975266 22160 consensus_queue.cc:260] T eb209814120a4e97a4fe05c18cd6d763 P 65b483c0d81042a89adcd922412a4e10 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "65b483c0d81042a89adcd922412a4e10" member_type: VOTER last_known_addr { host: "127.3.131.131" port: 39477 } } peers { permanent_uuid: "e62ab0b7280143c799337c11235d51cb" member_type: VOTER last_known_addr { host: "127.3.131.130" port: 36511 } } peers { permanent_uuid: "574266cc91594bd5a2465fb6ffb6f661" member_type: VOTER last_known_addr { host: "127.3.131.129" port: 41197 } }
I20260621 05:14:21.975535 22160 ts_tablet_manager.cc:1428] T eb209814120a4e97a4fe05c18cd6d763 P 65b483c0d81042a89adcd922412a4e10: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260621 05:14:21.975652 22139 heartbeater.cc:503] Master 127.3.131.190:32797 was elected leader, sending a full tablet report...
W20260621 05:14:22.011871 22006 tablet.cc:2369] T eb209814120a4e97a4fe05c18cd6d763 P e62ab0b7280143c799337c11235d51cb: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260621 05:14:22.042255 21872 tablet.cc:2369] T eb209814120a4e97a4fe05c18cd6d763 P 574266cc91594bd5a2465fb6ffb6f661: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260621 05:14:22.084385 22162 raft_consensus.cc:482] T eb209814120a4e97a4fe05c18cd6d763 P e62ab0b7280143c799337c11235d51cb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260621 05:14:22.084527 22162 raft_consensus.cc:504] T eb209814120a4e97a4fe05c18cd6d763 P e62ab0b7280143c799337c11235d51cb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "65b483c0d81042a89adcd922412a4e10" member_type: VOTER last_known_addr { host: "127.3.131.131" port: 39477 } } peers { permanent_uuid: "e62ab0b7280143c799337c11235d51cb" member_type: VOTER last_known_addr { host: "127.3.131.130" port: 36511 } } peers { permanent_uuid: "574266cc91594bd5a2465fb6ffb6f661" member_type: VOTER last_known_addr { host: "127.3.131.129" port: 41197 } }
I20260621 05:14:22.084981 22162 leader_election.cc:288] T eb209814120a4e97a4fe05c18cd6d763 P e62ab0b7280143c799337c11235d51cb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 65b483c0d81042a89adcd922412a4e10 (127.3.131.131:39477), 574266cc91594bd5a2465fb6ffb6f661 (127.3.131.129:41197)
I20260621 05:14:22.087949 22155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:14:22.085349 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.131.130:57595 (local address 127.3.131.129:41197)
0621 05:14:22.085396 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0621 05:14:22.085398 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:14:22.085408 (+ 10us) server_negotiation.cc:407] Connection header received
0621 05:14:22.085444 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:14:22.085445 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:14:22.085453 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:14:22.085531 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 05:14:22.085738 (+ 207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:14:22.086236 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:14:22.087212 (+ 976us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:14:22.087578 (+ 366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:14:22.087662 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:14:22.087671 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0621 05:14:22.087835 (+ 164us) server_negotiation.cc:299] Negotiation successful
0621 05:14:22.087880 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260621 05:14:22.087949 22007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:14:22.085209 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.131.129:41197 (local address 127.3.131.130:57595)
0621 05:14:22.085260 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0621 05:14:22.085273 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0621 05:14:22.085326 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:14:22.085536 (+ 210us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:14:22.085538 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:14:22.085540 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 05:14:22.085615 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:14:22.085617 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:14:22.086322 (+ 705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:14:22.086324 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:14:22.087077 (+ 753us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:14:22.087080 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:14:22.087660 (+ 580us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:14:22.087661 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:14:22.087745 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:14:22.087748 (+ 3us) client_negotiation.cc:769] Sending connection context
0621 05:14:22.087823 (+ 75us) client_negotiation.cc:240] Negotiation successful
0621 05:14:22.087868 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":29,"mutex_wait_us":1}
I20260621 05:14:22.088359 21825 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb209814120a4e97a4fe05c18cd6d763" candidate_uuid: "e62ab0b7280143c799337c11235d51cb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "574266cc91594bd5a2465fb6ffb6f661" is_pre_election: true
I20260621 05:14:22.088572 21825 raft_consensus.cc:2454] T eb209814120a4e97a4fe05c18cd6d763 P 574266cc91594bd5a2465fb6ffb6f661 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e62ab0b7280143c799337c11235d51cb in term 0.
I20260621 05:14:22.088706 22154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:14:22.085319 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.131.130:37109 (local address 127.3.131.131:39477)
0621 05:14:22.085367 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0621 05:14:22.085369 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:14:22.085499 (+ 130us) server_negotiation.cc:407] Connection header received
0621 05:14:22.085578 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:14:22.085579 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:14:22.085589 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:14:22.085670 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 05:14:22.085837 (+ 167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:14:22.086313 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:14:22.087775 (+ 1462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:14:22.088297 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:14:22.088400 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:14:22.088410 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0621 05:14:22.088593 (+ 183us) server_negotiation.cc:299] Negotiation successful
0621 05:14:22.088644 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260621 05:14:22.088712 22165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:14:22.085224 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.131.131:39477 (local address 127.3.131.130:37109)
0621 05:14:22.085439 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0621 05:14:22.085447 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0621 05:14:22.085521 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:14:22.085680 (+ 159us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:14:22.085681 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:14:22.085683 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 05:14:22.085761 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:14:22.085762 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:14:22.086399 (+ 637us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:14:22.086400 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:14:22.087683 (+ 1283us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:14:22.087686 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:14:22.088414 (+ 728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:14:22.088415 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:14:22.088522 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:14:22.088527 (+ 5us) client_negotiation.cc:769] Sending connection context
0621 05:14:22.088584 (+ 57us) client_negotiation.cc:240] Negotiation successful
0621 05:14:22.088624 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":202,"mutex_wait_us":16,"thread_start_us":132,"threads_started":1}
I20260621 05:14:22.088897 21895 leader_election.cc:302] T eb209814120a4e97a4fe05c18cd6d763 P e62ab0b7280143c799337c11235d51cb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 574266cc91594bd5a2465fb6ffb6f661, e62ab0b7280143c799337c11235d51cb; no voters:
I20260621 05:14:22.089037 22162 raft_consensus.cc:2781] T eb209814120a4e97a4fe05c18cd6d763 P e62ab0b7280143c799337c11235d51cb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260621 05:14:22.089120 22162 raft_consensus.cc:482] T eb209814120a4e97a4fe05c18cd6d763 P e62ab0b7280143c799337c11235d51cb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260621 05:14:22.089145 22162 raft_consensus.cc:3037] T eb209814120a4e97a4fe05c18cd6d763 P e62ab0b7280143c799337c11235d51cb [term 0 FOLLOWER]: Advancing to term 1
I20260621 05:14:22.089159 22093 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb209814120a4e97a4fe05c18cd6d763" candidate_uuid: "e62ab0b7280143c799337c11235d51cb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "65b483c0d81042a89adcd922412a4e10" is_pre_election: true
I20260621 05:14:22.089319 22093 raft_consensus.cc:2454] T eb209814120a4e97a4fe05c18cd6d763 P 65b483c0d81042a89adcd922412a4e10 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e62ab0b7280143c799337c11235d51cb in term 0.
I20260621 05:14:22.090040 22162 raft_consensus.cc:504] T eb209814120a4e97a4fe05c18cd6d763 P e62ab0b7280143c799337c11235d51cb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "65b483c0d81042a89adcd922412a4e10" member_type: VOTER last_known_addr { host: "127.3.131.131" port: 39477 } } peers { permanent_uuid: "e62ab0b7280143c799337c11235d51cb" member_type: VOTER last_known_addr { host: "127.3.131.130" port: 36511 } } peers { permanent_uuid: "574266cc91594bd5a2465fb6ffb6f661" member_type: VOTER last_known_addr { host: "127.3.131.129" port: 41197 } }
I20260621 05:14:22.090221 22162 leader_election.cc:288] T eb209814120a4e97a4fe05c18cd6d763 P e62ab0b7280143c799337c11235d51cb [CANDIDATE]: Term 1 election: Requested vote from peers 65b483c0d81042a89adcd922412a4e10 (127.3.131.131:39477), 574266cc91594bd5a2465fb6ffb6f661 (127.3.131.129:41197)
I20260621 05:14:22.090379 22093 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb209814120a4e97a4fe05c18cd6d763" candidate_uuid: "e62ab0b7280143c799337c11235d51cb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "65b483c0d81042a89adcd922412a4e10"
I20260621 05:14:22.090458 22093 raft_consensus.cc:3037] T eb209814120a4e97a4fe05c18cd6d763 P 65b483c0d81042a89adcd922412a4e10 [term 0 FOLLOWER]: Advancing to term 1
I20260621 05:14:22.090432 21825 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb209814120a4e97a4fe05c18cd6d763" candidate_uuid: "e62ab0b7280143c799337c11235d51cb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "574266cc91594bd5a2465fb6ffb6f661"
I20260621 05:14:22.090543 21825 raft_consensus.cc:3037] T eb209814120a4e97a4fe05c18cd6d763 P 574266cc91594bd5a2465fb6ffb6f661 [term 0 FOLLOWER]: Advancing to term 1
I20260621 05:14:22.091285 22093 raft_consensus.cc:2454] T eb209814120a4e97a4fe05c18cd6d763 P 65b483c0d81042a89adcd922412a4e10 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e62ab0b7280143c799337c11235d51cb in term 1.
I20260621 05:14:22.091387 21825 raft_consensus.cc:2454] T eb209814120a4e97a4fe05c18cd6d763 P 574266cc91594bd5a2465fb6ffb6f661 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e62ab0b7280143c799337c11235d51cb in term 1.
I20260621 05:14:22.091452 21894 leader_election.cc:302] T eb209814120a4e97a4fe05c18cd6d763 P e62ab0b7280143c799337c11235d51cb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 65b483c0d81042a89adcd922412a4e10, e62ab0b7280143c799337c11235d51cb; no voters:
I20260621 05:14:22.091580 22162 raft_consensus.cc:2781] T eb209814120a4e97a4fe05c18cd6d763 P e62ab0b7280143c799337c11235d51cb [term 1 FOLLOWER]: Leader election won for term 1
I20260621 05:14:22.091746 22162 raft_consensus.cc:686] T eb209814120a4e97a4fe05c18cd6d763 P e62ab0b7280143c799337c11235d51cb [term 1 LEADER]: Becoming Leader. State: Replica: e62ab0b7280143c799337c11235d51cb, State: Running, Role: LEADER
I20260621 05:14:22.091858 22162 consensus_queue.cc:237] T eb209814120a4e97a4fe05c18cd6d763 P e62ab0b7280143c799337c11235d51cb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "65b483c0d81042a89adcd922412a4e10" member_type: VOTER last_known_addr { host: "127.3.131.131" port: 39477 } } peers { permanent_uuid: "e62ab0b7280143c799337c11235d51cb" member_type: VOTER last_known_addr { host: "127.3.131.130" port: 36511 } } peers { permanent_uuid: "574266cc91594bd5a2465fb6ffb6f661" member_type: VOTER last_known_addr { host: "127.3.131.129" port: 41197 } }
I20260621 05:14:22.092859 21680 catalog_manager.cc:5495] T eb209814120a4e97a4fe05c18cd6d763 P e62ab0b7280143c799337c11235d51cb reported cstate change: term changed from 0 to 1, leader changed from <none> to e62ab0b7280143c799337c11235d51cb (127.3.131.130). New cstate: current_term: 1 leader_uuid: "e62ab0b7280143c799337c11235d51cb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "65b483c0d81042a89adcd922412a4e10" member_type: VOTER last_known_addr { host: "127.3.131.131" port: 39477 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e62ab0b7280143c799337c11235d51cb" member_type: VOTER last_known_addr { host: "127.3.131.130" port: 36511 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "574266cc91594bd5a2465fb6ffb6f661" member_type: VOTER last_known_addr { host: "127.3.131.129" port: 41197 } health_report { overall_health: UNKNOWN } } }
I20260621 05:14:22.111433 22152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:14:22.108610 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49984 (local address 127.3.131.130:36511)
0621 05:14:22.108675 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0621 05:14:22.108677 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:14:22.108687 (+ 10us) server_negotiation.cc:407] Connection header received
0621 05:14:22.108775 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:14:22.108776 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:14:22.108794 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:14:22.108874 (+ 80us) server_negotiation.cc:226] Negotiated authn=TOKEN
0621 05:14:22.109014 (+ 140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:14:22.109498 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:14:22.110514 (+ 1016us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:14:22.110778 (+ 264us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:14:22.110847 (+ 69us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:14:22.111001 (+ 154us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0621 05:14:22.111130 (+ 129us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0621 05:14:22.111285 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0621 05:14:22.111333 (+ 48us) server_negotiation.cc:299] Negotiation successful
0621 05:14:22.111345 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260621 05:14:22.113814 22093 raft_consensus.cc:1261] T eb209814120a4e97a4fe05c18cd6d763 P 65b483c0d81042a89adcd922412a4e10 [term 1 FOLLOWER]: Refusing update from remote peer e62ab0b7280143c799337c11235d51cb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 05:14:22.113816 21825 raft_consensus.cc:1261] T eb209814120a4e97a4fe05c18cd6d763 P 574266cc91594bd5a2465fb6ffb6f661 [term 1 FOLLOWER]: Refusing update from remote peer e62ab0b7280143c799337c11235d51cb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 05:14:22.114224 22166 consensus_queue.cc:1035] T eb209814120a4e97a4fe05c18cd6d763 P e62ab0b7280143c799337c11235d51cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "65b483c0d81042a89adcd922412a4e10" member_type: VOTER last_known_addr { host: "127.3.131.131" port: 39477 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 05:14:22.114359 22162 consensus_queue.cc:1035] T eb209814120a4e97a4fe05c18cd6d763 P e62ab0b7280143c799337c11235d51cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "574266cc91594bd5a2465fb6ffb6f661" member_type: VOTER last_known_addr { host: "127.3.131.129" port: 41197 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 05:14:22.117107 22169 mvcc.cc:205] Tried to move back new op lower bound from 7299149259213721600 to 7299149259128881152. Current Snapshot: MvccSnapshot[applied={T|T < 7299149259213721600}]
I20260621 05:14:22.118464 22170 mvcc.cc:205] Tried to move back new op lower bound from 7299149259213721600 to 7299149259128881152. Current Snapshot: MvccSnapshot[applied={T|T < 7299149259213721600 or (T in {7299149259213721600})}]
I20260621 05:14:22.120143 21679 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46450:
table { table_name: "test-table" } modify_external_catalogs: true
I20260621 05:14:22.120268 21679 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46450:
table { table_name: "test-table" } modify_external_catalogs: true
I20260621 05:14:22.121243 21679 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 65b67b67bb4e44dea10c37f4dd2af5ec: Sending DeleteTablet for 3 replicas of tablet eb209814120a4e97a4fe05c18cd6d763
I20260621 05:14:22.121634 21939 tablet_service.cc:1514] Processing DeleteTablet for tablet eb209814120a4e97a4fe05c18cd6d763 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 05:14:22 UTC) from {username='kudu', principal='kudu/127.3.131.190@KRBTEST.COM'} at 127.0.0.1:49976
I20260621 05:14:22.121721 22073 tablet_service.cc:1514] Processing DeleteTablet for tablet eb209814120a4e97a4fe05c18cd6d763 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 05:14:22 UTC) from {username='kudu', principal='kudu/127.3.131.190@KRBTEST.COM'} at 127.0.0.1:42364
I20260621 05:14:22.121762 21805 tablet_service.cc:1514] Processing DeleteTablet for tablet eb209814120a4e97a4fe05c18cd6d763 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 05:14:22 UTC) from {username='kudu', principal='kudu/127.3.131.190@KRBTEST.COM'} at 127.0.0.1:59440
I20260621 05:14:22.122042 22178 tablet_replica.cc:330] T eb209814120a4e97a4fe05c18cd6d763 P e62ab0b7280143c799337c11235d51cb: stopping tablet replica
I20260621 05:14:22.122138 3598 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWZOx2_/build/release/bin/kudu with pid 21742
I20260621 05:14:22.122246 22179 tablet_replica.cc:330] T eb209814120a4e97a4fe05c18cd6d763 P 65b483c0d81042a89adcd922412a4e10: stopping tablet replica
I20260621 05:14:22.122229 22178 raft_consensus.cc:2229] T eb209814120a4e97a4fe05c18cd6d763 P e62ab0b7280143c799337c11235d51cb [term 1 LEADER]: Raft consensus shutting down.
I20260621 05:14:22.122359 22179 raft_consensus.cc:2229] T eb209814120a4e97a4fe05c18cd6d763 P 65b483c0d81042a89adcd922412a4e10 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260621 05:14:22.122481 22179 raft_consensus.cc:2258] T eb209814120a4e97a4fe05c18cd6d763 P 65b483c0d81042a89adcd922412a4e10 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 05:14:22.122490 22178 raft_consensus.cc:2258] T eb209814120a4e97a4fe05c18cd6d763 P e62ab0b7280143c799337c11235d51cb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 05:14:22.123014 22179 ts_tablet_manager.cc:1905] T eb209814120a4e97a4fe05c18cd6d763 P 65b483c0d81042a89adcd922412a4e10: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 05:14:22.125077 22179 ts_tablet_manager.cc:1918] T eb209814120a4e97a4fe05c18cd6d763 P 65b483c0d81042a89adcd922412a4e10: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260621 05:14:22.125229 22179 log.cc:1196] T eb209814120a4e97a4fe05c18cd6d763 P 65b483c0d81042a89adcd922412a4e10: Deleting WAL directory at /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782018616648578-3598-0/minicluster-data/ts-2/wal/wals/eb209814120a4e97a4fe05c18cd6d763
I20260621 05:14:22.125581 22179 ts_tablet_manager.cc:1939] T eb209814120a4e97a4fe05c18cd6d763 P 65b483c0d81042a89adcd922412a4e10: Deleting consensus metadata
I20260621 05:14:22.125981 22178 ts_tablet_manager.cc:1905] T eb209814120a4e97a4fe05c18cd6d763 P e62ab0b7280143c799337c11235d51cb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 05:14:22.126142 21666 catalog_manager.cc:4841] TS 65b483c0d81042a89adcd922412a4e10 (127.3.131.131:39477): tablet eb209814120a4e97a4fe05c18cd6d763 (table test-table [id=6049ac0a904d40dfa96e5678016c7c75]) successfully deleted
I20260621 05:14:22.127837 22178 ts_tablet_manager.cc:1918] T eb209814120a4e97a4fe05c18cd6d763 P e62ab0b7280143c799337c11235d51cb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260621 05:14:22.127936 22178 log.cc:1196] T eb209814120a4e97a4fe05c18cd6d763 P e62ab0b7280143c799337c11235d51cb: Deleting WAL directory at /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782018616648578-3598-0/minicluster-data/ts-1/wal/wals/eb209814120a4e97a4fe05c18cd6d763
I20260621 05:14:22.128185 22178 ts_tablet_manager.cc:1939] T eb209814120a4e97a4fe05c18cd6d763 P e62ab0b7280143c799337c11235d51cb: Deleting consensus metadata
I20260621 05:14:22.128638 21667 catalog_manager.cc:4841] TS e62ab0b7280143c799337c11235d51cb (127.3.131.130:36511): tablet eb209814120a4e97a4fe05c18cd6d763 (table test-table [id=6049ac0a904d40dfa96e5678016c7c75]) successfully deleted
W20260621 05:14:22.129909 21667 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.3.131.129:41197 (error 108)
I20260621 05:14:22.130155 3598 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWZOx2_/build/release/bin/kudu with pid 21876
I20260621 05:14:22.131253 22151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:14:22.131071 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.131.129:41197 (local address 127.0.0.1:59456)
0621 05:14:22.131124 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0621 05:14:22.131211 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.131.129:41197: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
W20260621 05:14:22.131510 21667 catalog_manager.cc:4584] TS 574266cc91594bd5a2465fb6ffb6f661 (127.3.131.129:41197): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet eb209814120a4e97a4fe05c18cd6d763: Network error: Client connection negotiation failed: client connection to 127.3.131.129:41197: connect: Connection refused (error 111)
I20260621 05:14:22.135574 3598 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWZOx2_/build/release/bin/kudu with pid 22010
I20260621 05:14:22.140646 3598 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWZOx2_/build/release/bin/kudu with pid 21648
2026-06-21T05:14:22Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (2134 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-06-21T05:14:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-21T05:14:22Z Disabled control of system clock
I20260621 05:14:22.170171 3598 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWZOx2_/build/release/bin/kudu
/tmp/dist-test-taskWZOx2_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782018616648578-3598-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782018616648578-3598-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782018616648578-3598-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782018616648578-3598-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.131.190:44643
--webserver_interface=127.3.131.190
--webserver_port=0
--builtin_ntp_servers=127.3.131.148:41545
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.131.190:44643
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260621 05:14:22.254351 22185 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 05:14:22.254611 22185 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 05:14:22.254634 22185 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 05:14:22.256023 22185 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 05:14:22.256081 22185 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 05:14:22.256095 22185 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 05:14:22.256108 22185 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 05:14:22.256120 22185 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 05:14:22.257589 22185 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.131.148:41545
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782018616648578-3598-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782018616648578-3598-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.131.190:44643
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.131.190:44643
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782018616648578-3598-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.131.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22185
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782018616648578-3598-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 21 Jun 2026 04:16:16 UTC on ccycloud.quasar-xycaha.root.comops.site
build id 4457881
I20260621 05:14:22.257839 22185 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 05:14:22.258107 22185 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 05:14:22.263729 22193 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:14:22.263723 22191 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:14:22.263919 22185 server_base.cc:1034] running on GCE node
W20260621 05:14:22.263729 22190 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:14:22.264518 22185 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 05:14:22.264865 22185 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 05:14:22.266122 22185 hybrid_clock.cc:648] HybridClock initialized: now 1782018862266107 us; error 35 us; skew 500 ppm
I20260621 05:14:22.268221 22185 webserver.cc:460] Webserver started at http://127.3.131.190:38705/ using document root <none> and password file <none>
I20260621 05:14:22.268523 22185 fs_manager.cc:359] Metadata directory not provided
I20260621 05:14:22.268599 22185 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 05:14:22.268760 22185 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 05:14:22.269913 22185 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782018616648578-3598-0/minicluster-data/master-0/data/instance:
uuid: "c107cc5843fa43dfaa7df9a785c15a03"
format_stamp: "Formatted at 2026-06-21 05:14:22 on dist-test-slave-vrq1"
server_key: "e40bc1af0a52b9b4daf5922cf34b0717"
server_key_iv: "a09967c411f4537c01210d9110e49169"
server_key_version: "encryptionkey@0"
I20260621 05:14:22.270292 22185 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782018616648578-3598-0/minicluster-data/master-0/wal/instance:
uuid: "c107cc5843fa43dfaa7df9a785c15a03"
format_stamp: "Formatted at 2026-06-21 05:14:22 on dist-test-slave-vrq1"
server_key: "e40bc1af0a52b9b4daf5922cf34b0717"
server_key_iv: "a09967c411f4537c01210d9110e49169"
server_key_version: "encryptionkey@0"
I20260621 05:14:22.271780 22185 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260621 05:14:22.272696 22199 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 05:14:22.272958 22185 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260621 05:14:22.273074 22185 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782018616648578-3598-0/minicluster-data/master-0/data,/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782018616648578-3598-0/minicluster-data/master-0/wal
uuid: "c107cc5843fa43dfaa7df9a785c15a03"
format_stamp: "Formatted at 2026-06-21 05:14:22 on dist-test-slave-vrq1"
server_key: "e40bc1af0a52b9b4daf5922cf34b0717"
server_key_iv: "a09967c411f4537c01210d9110e49169"
server_key_version: "encryptionkey@0"
I20260621 05:14:22.273157 22185 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782018616648578-3598-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782018616648578-3598-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782018616648578-3598-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 05:14:22.284508 22185 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 05:14:22.284860 22185 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 05:14:22.285017 22185 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 05:14:22.289383 22185 rpc_server.cc:307] RPC server started. Bound to: 127.3.131.190:44643
I20260621 05:14:22.289459 22251 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.131.190:44643 every 8 connection(s)
I20260621 05:14:22.289827 22185 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782018616648578-3598-0/minicluster-data/master-0/data/info.pb
I20260621 05:14:22.290808 22252 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 05:14:22.293109 22252 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c107cc5843fa43dfaa7df9a785c15a03: Bootstrap starting.
I20260621 05:14:22.293650 22252 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c107cc5843fa43dfaa7df9a785c15a03: Neither blocks nor log segments found. Creating new log.
I20260621 05:14:22.293987 22252 log.cc:824] T 00000000000000000000000000000000 P c107cc5843fa43dfaa7df9a785c15a03: Log is configured to *not* fsync() on all Append() calls
I20260621 05:14:22.294631 22252 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c107cc5843fa43dfaa7df9a785c15a03: No bootstrap required, opened a new log
I20260621 05:14:22.295574 3598 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWZOx2_/build/release/bin/kudu as pid 22185
I20260621 05:14:22.295701 3598 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782018616648578-3598-0/minicluster-data/master-0/wal/instance
I20260621 05:14:22.295895 3598 external_mini_cluster.cc:1428] Setting key ce21eb852078939ef0dfb806d9612d3d
I20260621 05:14:22.296602 22252 raft_consensus.cc:348] T 00000000000000000000000000000000 P c107cc5843fa43dfaa7df9a785c15a03 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c107cc5843fa43dfaa7df9a785c15a03" member_type: VOTER last_known_addr { host: "127.3.131.190" port: 44643 } }
I20260621 05:14:22.296777 22252 raft_consensus.cc:374] T 00000000000000000000000000000000 P c107cc5843fa43dfaa7df9a785c15a03 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 05:14:22.296824 22252 raft_consensus.cc:729] T 00000000000000000000000000000000 P c107cc5843fa43dfaa7df9a785c15a03 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c107cc5843fa43dfaa7df9a785c15a03, State: Initialized, Role: FOLLOWER
I20260621 05:14:22.296948 22252 consensus_queue.cc:260] T 00000000000000000000000000000000 P c107cc5843fa43dfaa7df9a785c15a03 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c107cc5843fa43dfaa7df9a785c15a03" member_type: VOTER last_known_addr { host: "127.3.131.190" port: 44643 } }
I20260621 05:14:22.297035 22252 raft_consensus.cc:388] T 00000000000000000000000000000000 P c107cc5843fa43dfaa7df9a785c15a03 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260621 05:14:22.297070 22252 raft_consensus.cc:482] T 00000000000000000000000000000000 P c107cc5843fa43dfaa7df9a785c15a03 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260621 05:14:22.297120 22252 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c107cc5843fa43dfaa7df9a785c15a03 [term 0 FOLLOWER]: Advancing to term 1
I20260621 05:14:22.297694 22252 raft_consensus.cc:504] T 00000000000000000000000000000000 P c107cc5843fa43dfaa7df9a785c15a03 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c107cc5843fa43dfaa7df9a785c15a03" member_type: VOTER last_known_addr { host: "127.3.131.190" port: 44643 } }
I20260621 05:14:22.297784 22252 leader_election.cc:302] T 00000000000000000000000000000000 P c107cc5843fa43dfaa7df9a785c15a03 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c107cc5843fa43dfaa7df9a785c15a03; no voters:
I20260621 05:14:22.297958 22252 leader_election.cc:288] T 00000000000000000000000000000000 P c107cc5843fa43dfaa7df9a785c15a03 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260621 05:14:22.298003 22257 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c107cc5843fa43dfaa7df9a785c15a03 [term 1 FOLLOWER]: Leader election won for term 1
I20260621 05:14:22.298113 22257 raft_consensus.cc:686] T 00000000000000000000000000000000 P c107cc5843fa43dfaa7df9a785c15a03 [term 1 LEADER]: Becoming Leader. State: Replica: c107cc5843fa43dfaa7df9a785c15a03, State: Running, Role: LEADER
I20260621 05:14:22.298192 22257 consensus_queue.cc:237] T 00000000000000000000000000000000 P c107cc5843fa43dfaa7df9a785c15a03 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c107cc5843fa43dfaa7df9a785c15a03" member_type: VOTER last_known_addr { host: "127.3.131.190" port: 44643 } }
I20260621 05:14:22.298233 22252 sys_catalog.cc:564] T 00000000000000000000000000000000 P c107cc5843fa43dfaa7df9a785c15a03 [sys.catalog]: configured and running, proceeding with master startup.
I20260621 05:14:22.298559 22257 sys_catalog.cc:455] T 00000000000000000000000000000000 P c107cc5843fa43dfaa7df9a785c15a03 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c107cc5843fa43dfaa7df9a785c15a03. Latest consensus state: current_term: 1 leader_uuid: "c107cc5843fa43dfaa7df9a785c15a03" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c107cc5843fa43dfaa7df9a785c15a03" member_type: VOTER last_known_addr { host: "127.3.131.190" port: 44643 } } }
I20260621 05:14:22.298575 22258 sys_catalog.cc:455] T 00000000000000000000000000000000 P c107cc5843fa43dfaa7df9a785c15a03 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c107cc5843fa43dfaa7df9a785c15a03" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c107cc5843fa43dfaa7df9a785c15a03" member_type: VOTER last_known_addr { host: "127.3.131.190" port: 44643 } } }
I20260621 05:14:22.298684 22258 sys_catalog.cc:458] T 00000000000000000000000000000000 P c107cc5843fa43dfaa7df9a785c15a03 [sys.catalog]: This master's current role is: LEADER
I20260621 05:14:22.298679 22257 sys_catalog.cc:458] T 00000000000000000000000000000000 P c107cc5843fa43dfaa7df9a785c15a03 [sys.catalog]: This master's current role is: LEADER
I20260621 05:14:22.298978 22263 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260621 05:14:22.299521 22263 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260621 05:14:22.302271 22256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:14:22.296462 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41904 (local address 127.3.131.190:44643)
0621 05:14:22.296673 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0621 05:14:22.296681 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0621 05:14:22.296705 (+ 24us) server_negotiation.cc:407] Connection header received
0621 05:14:22.296815 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:14:22.296822 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:14:22.296895 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:14:22.297050 (+ 155us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:14:22.297264 (+ 214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:14:22.298841 (+ 1577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:14:22.300098 (+ 1257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:14:22.300704 (+ 606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:14:22.300815 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:14:22.301559 (+ 744us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:14:22.301563 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:14:22.301564 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 05:14:22.301770 (+ 206us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:14:22.301811 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 05:14:22.301826 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:14:22.301899 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0621 05:14:22.301967 (+ 68us) server_negotiation.cc:299] Negotiation successful
0621 05:14:22.302054 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":134,"thread_start_us":98,"threads_started":1}
I20260621 05:14:22.307684 22263 catalog_manager.cc:1347] Generated new cluster ID: cc1ffab9b0f9436d86355758a819cc7d
I20260621 05:14:22.307786 22263 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260621 05:14:22.320690 22263 catalog_manager.cc:1370] Generated new certificate authority record
I20260621 05:14:22.321456 22263 catalog_manager.cc:1504] Loading token signing keys...
I20260621 05:14:22.327729 22263 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c107cc5843fa43dfaa7df9a785c15a03: Generated new TSK 0
I20260621 05:14:22.327960 22263 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260621 05:14:22.335143 3598 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWZOx2_/build/release/bin/kudu
/tmp/dist-test-taskWZOx2_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782018616648578-3598-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782018616648578-3598-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782018616648578-3598-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782018616648578-3598-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.131.129:0
--local_ip_for_outbound_sockets=127.3.131.129
--webserver_interface=127.3.131.129
--webserver_port=0
--tserver_master_addrs=127.3.131.190:44643
--builtin_ntp_servers=127.3.131.148:41545
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260621 05:14:22.429292 22276 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 05:14:22.429536 22276 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 05:14:22.429560 22276 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 05:14:22.430886 22276 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 05:14:22.430929 22276 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 05:14:22.430958 22276 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.131.129
I20260621 05:14:22.432430 22276 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.131.148:41545
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782018616648578-3598-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782018616648578-3598-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.131.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782018616648578-3598-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.131.129
--webserver_port=0
--tserver_master_addrs=127.3.131.190:44643
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22276
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.131.129
--log_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782018616648578-3598-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 21 Jun 2026 04:16:16 UTC on ccycloud.quasar-xycaha.root.comops.site
build id 4457881
I20260621 05:14:22.432725 22276 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 05:14:22.433048 22276 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 05:14:22.438504 22284 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:14:22.438512 22281 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:14:22.438752 22276 server_base.cc:1034] running on GCE node
W20260621 05:14:22.438532 22282 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:14:22.439070 22276 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 05:14:22.439368 22276 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 05:14:22.440536 22276 hybrid_clock.cc:648] HybridClock initialized: now 1782018862440521 us; error 33 us; skew 500 ppm
I20260621 05:14:22.441955 22276 webserver.cc:460] Webserver started at http://127.3.131.129:34211/ using document root <none> and password file <none>
I20260621 05:14:22.442219 22276 fs_manager.cc:359] Metadata directory not provided
I20260621 05:14:22.442296 22276 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 05:14:22.442440 22276 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 05:14:22.443629 22276 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782018616648578-3598-0/minicluster-data/ts-0/data/instance:
uuid: "262ca76d50824832b41b868a4b3c1560"
format_stamp: "Formatted at 2026-06-21 05:14:22 on dist-test-slave-vrq1"
server_key: "bef2a362f2e3b9311ad1493af8a6c8ce"
server_key_iv: "d39736aff439f6d9b0ddd1753ef2aa7f"
server_key_version: "encryptionkey@0"
I20260621 05:14:22.444110 22276 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782018616648578-3598-0/minicluster-data/ts-0/wal/instance:
uuid: "262ca76d50824832b41b868a4b3c1560"
format_stamp: "Formatted at 2026-06-21 05:14:22 on dist-test-slave-vrq1"
server_key: "bef2a362f2e3b9311ad1493af8a6c8ce"
server_key_iv: "d39736aff439f6d9b0ddd1753ef2aa7f"
server_key_version: "encryptionkey@0"
I20260621 05:14:22.445588 22276 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260621 05:14:22.446440 22290 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 05:14:22.446645 22276 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260621 05:14:22.446789 22276 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782018616648578-3598-0/minicluster-data/ts-0/data,/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782018616648578-3598-0/minicluster-data/ts-0/wal
uuid: "262ca76d50824832b41b868a4b3c1560"
format_stamp: "Formatted at 2026-06-21 05:14:22 on dist-test-slave-vrq1"
server_key: "bef2a362f2e3b9311ad1493af8a6c8ce"
server_key_iv: "d39736aff439f6d9b0ddd1753ef2aa7f"
server_key_version: "encryptionkey@0"
I20260621 05:14:22.446874 22276 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782018616648578-3598-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782018616648578-3598-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782018616648578-3598-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 05:14:22.455798 22276 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 05:14:22.456147 22276 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 05:14:22.456318 22276 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 05:14:22.456568 22276 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 05:14:22.456920 22276 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 05:14:22.456971 22276 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 05:14:22.457008 22276 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 05:14:22.457034 22276 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 05:14:22.463315 22276 rpc_server.cc:307] RPC server started. Bound to: 127.3.131.129:33575
I20260621 05:14:22.463392 22403 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.131.129:33575 every 8 connection(s)
I20260621 05:14:22.463752 22276 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782018616648578-3598-0/minicluster-data/ts-0/data/info.pb
I20260621 05:14:22.469623 22256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:14:22.464580 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.131.129:45135 (local address 127.3.131.190:44643)
0621 05:14:22.464630 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0621 05:14:22.464633 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0621 05:14:22.464904 (+ 271us) server_negotiation.cc:407] Connection header received
0621 05:14:22.465420 (+ 516us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:14:22.465421 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:14:22.465435 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:14:22.465504 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:14:22.466231 (+ 727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:14:22.466776 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:14:22.467913 (+ 1137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:14:22.468120 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:14:22.468183 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:14:22.469248 (+ 1065us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:14:22.469249 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:14:22.469249 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 05:14:22.469255 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:14:22.469269 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 05:14:22.469274 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:14:22.469320 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0621 05:14:22.469522 (+ 202us) server_negotiation.cc:299] Negotiation successful
0621 05:14:22.469556 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260621 05:14:22.469732 22406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:14:22.464540 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.131.190:44643 (local address 127.3.131.129:45135)
0621 05:14:22.464738 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0621 05:14:22.464782 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0621 05:14:22.465311 (+ 529us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:14:22.465578 (+ 267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:14:22.465582 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:14:22.465590 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:14:22.466102 (+ 512us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:14:22.466105 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:14:22.466890 (+ 785us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:14:22.466891 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:14:22.467824 (+ 933us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:14:22.467827 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:14:22.468242 (+ 415us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:14:22.468243 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:14:22.468388 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:14:22.469096 (+ 708us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0621 05:14:22.469096 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:14:22.469128 (+ 32us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0621 05:14:22.469155 (+ 27us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 05:14:22.469351 (+ 196us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 05:14:22.469358 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 05:14:22.469362 (+ 4us) client_negotiation.cc:769] Sending connection context
0621 05:14:22.469454 (+ 92us) client_negotiation.cc:240] Negotiation successful
0621 05:14:22.469552 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":139,"threads_started":1}
I20260621 05:14:22.470352 22404 heartbeater.cc:347] Connected to a master server at 127.3.131.190:44643
I20260621 05:14:22.470479 22404 heartbeater.cc:464] Registering TS with master...
I20260621 05:14:22.470718 22404 heartbeater.cc:511] Master 127.3.131.190:44643 requested a full tablet report, sending...
I20260621 05:14:22.471312 3598 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWZOx2_/build/release/bin/kudu as pid 22276
I20260621 05:14:22.471415 3598 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782018616648578-3598-0/minicluster-data/ts-0/wal/instance
I20260621 05:14:22.471596 3598 external_mini_cluster.cc:1428] Setting key 94d88948d8c9931b30fb6310d28ce2e4
I20260621 05:14:22.471613 22216 ts_manager.cc:194] Registered new tserver with Master: 262ca76d50824832b41b868a4b3c1560 (127.3.131.129:33575)
I20260621 05:14:22.472770 22216 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.3.131.129:45135
I20260621 05:14:22.472817 3598 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWZOx2_/build/release/bin/kudu
/tmp/dist-test-taskWZOx2_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782018616648578-3598-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782018616648578-3598-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782018616648578-3598-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782018616648578-3598-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.131.130:0
--local_ip_for_outbound_sockets=127.3.131.130
--webserver_interface=127.3.131.130
--webserver_port=0
--tserver_master_addrs=127.3.131.190:44643
--builtin_ntp_servers=127.3.131.148:41545
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260621 05:14:22.560320 22407 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 05:14:22.560559 22407 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 05:14:22.560583 22407 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 05:14:22.561885 22407 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 05:14:22.561937 22407 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 05:14:22.561967 22407 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.131.130
I20260621 05:14:22.563540 22407 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.131.148:41545
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782018616648578-3598-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782018616648578-3598-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.131.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782018616648578-3598-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.131.130
--webserver_port=0
--tserver_master_addrs=127.3.131.190:44643
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22407
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.131.130
--log_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782018616648578-3598-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 21 Jun 2026 04:16:16 UTC on ccycloud.quasar-xycaha.root.comops.site
build id 4457881
I20260621 05:14:22.563829 22407 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 05:14:22.564162 22407 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 05:14:22.570114 22415 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:14:22.570261 22407 server_base.cc:1034] running on GCE node
W20260621 05:14:22.570137 22413 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:14:22.570133 22412 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:14:22.570595 22407 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 05:14:22.570904 22407 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 05:14:22.572099 22407 hybrid_clock.cc:648] HybridClock initialized: now 1782018862572075 us; error 91 us; skew 500 ppm
I20260621 05:14:22.573601 22407 webserver.cc:460] Webserver started at http://127.3.131.130:45177/ using document root <none> and password file <none>
I20260621 05:14:22.573881 22407 fs_manager.cc:359] Metadata directory not provided
I20260621 05:14:22.573932 22407 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 05:14:22.574100 22407 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 05:14:22.575450 22407 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782018616648578-3598-0/minicluster-data/ts-1/data/instance:
uuid: "9ddc0e22632743cb9a8a9f18596cb345"
format_stamp: "Formatted at 2026-06-21 05:14:22 on dist-test-slave-vrq1"
server_key: "16566d942034b4c534c6c196740fb313"
server_key_iv: "d08a50085e3fbaa82c131c1c192cf093"
server_key_version: "encryptionkey@0"
I20260621 05:14:22.575846 22407 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782018616648578-3598-0/minicluster-data/ts-1/wal/instance:
uuid: "9ddc0e22632743cb9a8a9f18596cb345"
format_stamp: "Formatted at 2026-06-21 05:14:22 on dist-test-slave-vrq1"
server_key: "16566d942034b4c534c6c196740fb313"
server_key_iv: "d08a50085e3fbaa82c131c1c192cf093"
server_key_version: "encryptionkey@0"
I20260621 05:14:22.577605 22407 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.002s sys 0.000s
I20260621 05:14:22.578745 22421 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 05:14:22.579030 22407 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260621 05:14:22.579120 22407 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782018616648578-3598-0/minicluster-data/ts-1/data,/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782018616648578-3598-0/minicluster-data/ts-1/wal
uuid: "9ddc0e22632743cb9a8a9f18596cb345"
format_stamp: "Formatted at 2026-06-21 05:14:22 on dist-test-slave-vrq1"
server_key: "16566d942034b4c534c6c196740fb313"
server_key_iv: "d08a50085e3fbaa82c131c1c192cf093"
server_key_version: "encryptionkey@0"
I20260621 05:14:22.579244 22407 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782018616648578-3598-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782018616648578-3598-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782018616648578-3598-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 05:14:22.591038 22407 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 05:14:22.591452 22407 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 05:14:22.591602 22407 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 05:14:22.591861 22407 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 05:14:22.592223 22407 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 05:14:22.592276 22407 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 05:14:22.592321 22407 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 05:14:22.592347 22407 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 05:14:22.599745 22407 rpc_server.cc:307] RPC server started. Bound to: 127.3.131.130:46557
I20260621 05:14:22.599828 22534 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.131.130:46557 every 8 connection(s)
I20260621 05:14:22.600212 22407 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782018616648578-3598-0/minicluster-data/ts-1/data/info.pb
I20260621 05:14:22.606341 22256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:14:22.601033 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.131.130:38377 (local address 127.3.131.190:44643)
0621 05:14:22.601102 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0621 05:14:22.601105 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0621 05:14:22.601404 (+ 299us) server_negotiation.cc:407] Connection header received
0621 05:14:22.601923 (+ 519us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:14:22.601924 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:14:22.601935 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:14:22.602001 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:14:22.603109 (+ 1108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:14:22.603702 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:14:22.604790 (+ 1088us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:14:22.604996 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:14:22.605116 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:14:22.606009 (+ 893us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:14:22.606011 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:14:22.606011 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 05:14:22.606016 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:14:22.606033 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 05:14:22.606038 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:14:22.606090 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0621 05:14:22.606241 (+ 151us) server_negotiation.cc:299] Negotiation successful
0621 05:14:22.606271 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260621 05:14:22.606475 22537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:14:22.601009 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.131.190:44643 (local address 127.3.131.130:38377)
0621 05:14:22.601239 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0621 05:14:22.601283 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0621 05:14:22.601812 (+ 529us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:14:22.602104 (+ 292us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:14:22.602108 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:14:22.602117 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:14:22.602996 (+ 879us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:14:22.603003 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:14:22.603797 (+ 794us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:14:22.603798 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:14:22.604691 (+ 893us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:14:22.604694 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:14:22.605096 (+ 402us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:14:22.605097 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:14:22.605244 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:14:22.605850 (+ 606us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0621 05:14:22.605850 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:14:22.605882 (+ 32us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0621 05:14:22.605909 (+ 27us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 05:14:22.606098 (+ 189us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 05:14:22.606102 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 05:14:22.606104 (+ 2us) client_negotiation.cc:769] Sending connection context
0621 05:14:22.606183 (+ 79us) client_negotiation.cc:240] Negotiation successful
0621 05:14:22.606272 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":161,"thread_start_us":95,"threads_started":1}
I20260621 05:14:22.607041 22535 heartbeater.cc:347] Connected to a master server at 127.3.131.190:44643
I20260621 05:14:22.607198 22535 heartbeater.cc:464] Registering TS with master...
I20260621 05:14:22.607419 22535 heartbeater.cc:511] Master 127.3.131.190:44643 requested a full tablet report, sending...
I20260621 05:14:22.607842 22216 ts_manager.cc:194] Registered new tserver with Master: 9ddc0e22632743cb9a8a9f18596cb345 (127.3.131.130:46557)
I20260621 05:14:22.608278 22216 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.3.131.130:38377
I20260621 05:14:22.608839 3598 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWZOx2_/build/release/bin/kudu as pid 22407
I20260621 05:14:22.608937 3598 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782018616648578-3598-0/minicluster-data/ts-1/wal/instance
I20260621 05:14:22.609139 3598 external_mini_cluster.cc:1428] Setting key 3c7c47be0a1e9eef1eecebbc5e259939
I20260621 05:14:22.610471 3598 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWZOx2_/build/release/bin/kudu
/tmp/dist-test-taskWZOx2_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782018616648578-3598-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782018616648578-3598-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782018616648578-3598-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782018616648578-3598-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.131.131:0
--local_ip_for_outbound_sockets=127.3.131.131
--webserver_interface=127.3.131.131
--webserver_port=0
--tserver_master_addrs=127.3.131.190:44643
--builtin_ntp_servers=127.3.131.148:41545
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260621 05:14:22.696915 22538 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 05:14:22.697150 22538 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 05:14:22.697175 22538 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 05:14:22.698475 22538 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 05:14:22.698518 22538 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 05:14:22.698608 22538 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.131.131
I20260621 05:14:22.700196 22538 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.131.148:41545
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782018616648578-3598-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782018616648578-3598-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.131.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782018616648578-3598-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.131.131
--webserver_port=0
--tserver_master_addrs=127.3.131.190:44643
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22538
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.131.131
--log_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782018616648578-3598-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 21 Jun 2026 04:16:16 UTC on ccycloud.quasar-xycaha.root.comops.site
build id 4457881
I20260621 05:14:22.700474 22538 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 05:14:22.700812 22538 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 05:14:22.706194 22543 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:14:22.706194 22546 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:14:22.706204 22544 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:14:22.706619 22538 server_base.cc:1034] running on GCE node
I20260621 05:14:22.706842 22538 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 05:14:22.707085 22538 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 05:14:22.708292 22538 hybrid_clock.cc:648] HybridClock initialized: now 1782018862708282 us; error 50 us; skew 500 ppm
I20260621 05:14:22.709672 22538 webserver.cc:460] Webserver started at http://127.3.131.131:37889/ using document root <none> and password file <none>
I20260621 05:14:22.709936 22538 fs_manager.cc:359] Metadata directory not provided
I20260621 05:14:22.709986 22538 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 05:14:22.710114 22538 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 05:14:22.711344 22538 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782018616648578-3598-0/minicluster-data/ts-2/data/instance:
uuid: "c8590a7e27df4a6991e8c5be762b6ca0"
format_stamp: "Formatted at 2026-06-21 05:14:22 on dist-test-slave-vrq1"
server_key: "8db3a4dd5ea40167f1cd4fc97fe7c2ee"
server_key_iv: "3c73f08ba3f23e67501956e5302a090e"
server_key_version: "encryptionkey@0"
I20260621 05:14:22.711715 22538 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782018616648578-3598-0/minicluster-data/ts-2/wal/instance:
uuid: "c8590a7e27df4a6991e8c5be762b6ca0"
format_stamp: "Formatted at 2026-06-21 05:14:22 on dist-test-slave-vrq1"
server_key: "8db3a4dd5ea40167f1cd4fc97fe7c2ee"
server_key_iv: "3c73f08ba3f23e67501956e5302a090e"
server_key_version: "encryptionkey@0"
I20260621 05:14:22.713155 22538 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260621 05:14:22.714108 22552 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 05:14:22.714421 22538 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260621 05:14:22.714511 22538 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782018616648578-3598-0/minicluster-data/ts-2/data,/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782018616648578-3598-0/minicluster-data/ts-2/wal
uuid: "c8590a7e27df4a6991e8c5be762b6ca0"
format_stamp: "Formatted at 2026-06-21 05:14:22 on dist-test-slave-vrq1"
server_key: "8db3a4dd5ea40167f1cd4fc97fe7c2ee"
server_key_iv: "3c73f08ba3f23e67501956e5302a090e"
server_key_version: "encryptionkey@0"
I20260621 05:14:22.714671 22538 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782018616648578-3598-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782018616648578-3598-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782018616648578-3598-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 05:14:22.732591 22538 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 05:14:22.733031 22538 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 05:14:22.733246 22538 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 05:14:22.733527 22538 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 05:14:22.733901 22538 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 05:14:22.733955 22538 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 05:14:22.733992 22538 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 05:14:22.734018 22538 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 05:14:22.740195 22538 rpc_server.cc:307] RPC server started. Bound to: 127.3.131.131:41949
I20260621 05:14:22.740319 22665 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.131.131:41949 every 8 connection(s)
I20260621 05:14:22.740610 22538 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782018616648578-3598-0/minicluster-data/ts-2/data/info.pb
I20260621 05:14:22.746294 22256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:14:22.741479 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.131.131:37455 (local address 127.3.131.190:44643)
0621 05:14:22.741558 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0621 05:14:22.741560 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:14:22.741757 (+ 197us) server_negotiation.cc:407] Connection header received
0621 05:14:22.742396 (+ 639us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:14:22.742397 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:14:22.742408 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:14:22.742464 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:14:22.743168 (+ 704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:14:22.743762 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:14:22.744814 (+ 1052us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:14:22.745010 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:14:22.745082 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:14:22.745988 (+ 906us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:14:22.745989 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:14:22.745989 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 05:14:22.745994 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:14:22.746008 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 05:14:22.746012 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:14:22.746060 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0621 05:14:22.746198 (+ 138us) server_negotiation.cc:299] Negotiation successful
0621 05:14:22.746230 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260621 05:14:22.746318 3598 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWZOx2_/build/release/bin/kudu as pid 22538
I20260621 05:14:22.746505 3598 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782018616648578-3598-0/minicluster-data/ts-2/wal/instance
I20260621 05:14:22.746574 22668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:14:22.741409 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.131.190:44643 (local address 127.3.131.131:37455)
0621 05:14:22.741579 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0621 05:14:22.741614 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0621 05:14:22.742103 (+ 489us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:14:22.742531 (+ 428us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:14:22.742535 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:14:22.742543 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:14:22.743064 (+ 521us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:14:22.743068 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:14:22.743870 (+ 802us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:14:22.743870 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:14:22.744720 (+ 850us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:14:22.744722 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:14:22.745111 (+ 389us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:14:22.745112 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:14:22.745243 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:14:22.745828 (+ 585us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0621 05:14:22.745828 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:14:22.745859 (+ 31us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0621 05:14:22.745883 (+ 24us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 05:14:22.746112 (+ 229us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 05:14:22.746118 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 05:14:22.746121 (+ 3us) client_negotiation.cc:769] Sending connection context
0621 05:14:22.746195 (+ 74us) client_negotiation.cc:240] Negotiation successful
0621 05:14:22.746317 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":62,"threads_started":1}
I20260621 05:14:22.746711 3598 external_mini_cluster.cc:1428] Setting key a7998ef7748e2b4ddbe765e355cde8c4
I20260621 05:14:22.747366 22666 heartbeater.cc:347] Connected to a master server at 127.3.131.190:44643
I20260621 05:14:22.747483 22666 heartbeater.cc:464] Registering TS with master...
I20260621 05:14:22.747675 22666 heartbeater.cc:511] Master 127.3.131.190:44643 requested a full tablet report, sending...
I20260621 05:14:22.748036 22216 ts_manager.cc:194] Registered new tserver with Master: c8590a7e27df4a6991e8c5be762b6ca0 (127.3.131.131:41949)
I20260621 05:14:22.748382 22216 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.3.131.131:37455
I20260621 05:14:22.759268 3598 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260621 05:14:22.765089 22256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:14:22.760978 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41914 (local address 127.3.131.190:44643)
0621 05:14:22.761057 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0621 05:14:22.761059 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:14:22.761087 (+ 28us) server_negotiation.cc:407] Connection header received
0621 05:14:22.761380 (+ 293us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:14:22.761381 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:14:22.761394 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:14:22.761471 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:14:22.761703 (+ 232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:14:22.762229 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:14:22.763045 (+ 816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:14:22.763289 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:14:22.763355 (+ 66us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:14:22.764487 (+ 1132us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:14:22.764488 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:14:22.764488 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 05:14:22.764494 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:14:22.764509 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 05:14:22.764513 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:14:22.764607 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0621 05:14:22.764950 (+ 343us) server_negotiation.cc:299] Negotiation successful
0621 05:14:22.764967 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260621 05:14:22.767482 22216 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41914:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260621 05:14:22.769129 22216 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260621 05:14:22.775759 22679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:14:22.771783 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33174 (local address 127.3.131.129:33575)
0621 05:14:22.771994 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0621 05:14:22.771999 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 05:14:22.772008 (+ 9us) server_negotiation.cc:407] Connection header received
0621 05:14:22.772044 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:14:22.772046 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:14:22.772059 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:14:22.772300 (+ 241us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 05:14:22.772362 (+ 62us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:14:22.773155 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:14:22.774556 (+ 1401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:14:22.775326 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:14:22.775409 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:14:22.775548 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0621 05:14:22.775577 (+ 29us) server_negotiation.cc:299] Negotiation successful
0621 05:14:22.775682 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":182,"thread_start_us":100,"threads_started":1}
I20260621 05:14:22.775745 22675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:14:22.771421 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.131.129:33575 (local address 127.0.0.1:33174)
0621 05:14:22.771828 (+ 407us) negotiation.cc:107] Waiting for socket to connect
0621 05:14:22.771837 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0621 05:14:22.771871 (+ 34us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:14:22.772169 (+ 298us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:14:22.772170 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:14:22.772172 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 05:14:22.772220 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:14:22.772222 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:14:22.773242 (+ 1020us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:14:22.773243 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:14:22.774469 (+ 1226us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:14:22.774472 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:14:22.775408 (+ 936us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:14:22.775408 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:14:22.775485 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:14:22.775499 (+ 14us) client_negotiation.cc:769] Sending connection context
0621 05:14:22.775560 (+ 61us) client_negotiation.cc:240] Negotiation successful
0621 05:14:22.775654 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":393,"mutex_wait_us":3,"thread_start_us":90,"threads_started":1}
I20260621 05:14:22.775928 22674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:14:22.771400 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.131.130:46557 (local address 127.0.0.1:47442)
0621 05:14:22.771524 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0621 05:14:22.771565 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0621 05:14:22.771634 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:14:22.772231 (+ 597us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:14:22.772232 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:14:22.772234 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 05:14:22.772271 (+ 37us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:14:22.772272 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:14:22.773629 (+ 1357us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:14:22.773630 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:14:22.774688 (+ 1058us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:14:22.774689 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:14:22.775681 (+ 992us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:14:22.775682 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:14:22.775746 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:14:22.775751 (+ 5us) client_negotiation.cc:769] Sending connection context
0621 05:14:22.775784 (+ 33us) client_negotiation.cc:240] Negotiation successful
0621 05:14:22.775836 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":99,"mutex_wait_us":6,"thread_start_us":75,"threads_started":1}
I20260621 05:14:22.776254 22676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:14:22.771405 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.131.131:41949 (local address 127.0.0.1:42330)
0621 05:14:22.771673 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0621 05:14:22.771685 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0621 05:14:22.771739 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:14:22.772127 (+ 388us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:14:22.772130 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:14:22.772133 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 05:14:22.772214 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:14:22.772217 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:14:22.773631 (+ 1414us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:14:22.773632 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:14:22.774725 (+ 1093us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:14:22.774727 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:14:22.775704 (+ 977us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:14:22.775705 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:14:22.775754 (+ 49us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:14:22.775757 (+ 3us) client_negotiation.cc:769] Sending connection context
0621 05:14:22.775784 (+ 27us) client_negotiation.cc:240] Negotiation successful
0621 05:14:22.775887 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":247,"mutex_wait_us":114,"thread_start_us":106,"threads_started":1}
I20260621 05:14:22.776358 22678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:14:22.771727 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42330 (local address 127.3.131.131:41949)
0621 05:14:22.771924 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0621 05:14:22.771929 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 05:14:22.771941 (+ 12us) server_negotiation.cc:407] Connection header received
0621 05:14:22.771985 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:14:22.771987 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:14:22.772005 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:14:22.772107 (+ 102us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 05:14:22.772303 (+ 196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:14:22.773507 (+ 1204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:14:22.774830 (+ 1323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:14:22.775567 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:14:22.775973 (+ 406us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:14:22.776180 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0621 05:14:22.776212 (+ 32us) server_negotiation.cc:299] Negotiation successful
0621 05:14:22.776256 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":154,"thread_start_us":96,"threads_started":1}
I20260621 05:14:22.776769 22338 tablet_service.cc:1467] Processing CreateTablet for tablet 3d05b8c967dc4764a50fde0f509797c7 (DEFAULT_TABLE table=test-table [id=459918e4a5424cf3bec26e57ee0fa7ce]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 05:14:22.777004 22677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:14:22.771580 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47442 (local address 127.3.131.130:46557)
0621 05:14:22.771985 (+ 405us) server_negotiation.cc:206] Beginning negotiation
0621 05:14:22.771989 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 05:14:22.771997 (+ 8us) server_negotiation.cc:407] Connection header received
0621 05:14:22.772029 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:14:22.772030 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:14:22.772042 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:14:22.772349 (+ 307us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 05:14:22.772410 (+ 61us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:14:22.773543 (+ 1133us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:14:22.774769 (+ 1226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:14:22.775567 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:14:22.776511 (+ 944us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:14:22.776845 (+ 334us) server_negotiation.cc:1035] Waiting for connection context
0621 05:14:22.776872 (+ 27us) server_negotiation.cc:299] Negotiation successful
0621 05:14:22.776916 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":363,"thread_start_us":91,"threads_started":1}
I20260621 05:14:22.776983 22600 tablet_service.cc:1467] Processing CreateTablet for tablet 3d05b8c967dc4764a50fde0f509797c7 (DEFAULT_TABLE table=test-table [id=459918e4a5424cf3bec26e57ee0fa7ce]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 05:14:22.777196 22338 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3d05b8c967dc4764a50fde0f509797c7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 05:14:22.777343 22600 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3d05b8c967dc4764a50fde0f509797c7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 05:14:22.779979 22680 tablet_bootstrap.cc:492] T 3d05b8c967dc4764a50fde0f509797c7 P c8590a7e27df4a6991e8c5be762b6ca0: Bootstrap starting.
I20260621 05:14:22.780293 22681 tablet_bootstrap.cc:492] T 3d05b8c967dc4764a50fde0f509797c7 P 262ca76d50824832b41b868a4b3c1560: Bootstrap starting.
I20260621 05:14:22.780541 22469 tablet_service.cc:1467] Processing CreateTablet for tablet 3d05b8c967dc4764a50fde0f509797c7 (DEFAULT_TABLE table=test-table [id=459918e4a5424cf3bec26e57ee0fa7ce]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 05:14:22.780723 22680 tablet_bootstrap.cc:654] T 3d05b8c967dc4764a50fde0f509797c7 P c8590a7e27df4a6991e8c5be762b6ca0: Neither blocks nor log segments found. Creating new log.
I20260621 05:14:22.780880 22469 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3d05b8c967dc4764a50fde0f509797c7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 05:14:22.780933 22681 tablet_bootstrap.cc:654] T 3d05b8c967dc4764a50fde0f509797c7 P 262ca76d50824832b41b868a4b3c1560: Neither blocks nor log segments found. Creating new log.
I20260621 05:14:22.781064 22680 log.cc:824] T 3d05b8c967dc4764a50fde0f509797c7 P c8590a7e27df4a6991e8c5be762b6ca0: Log is configured to *not* fsync() on all Append() calls
I20260621 05:14:22.781250 22681 log.cc:824] T 3d05b8c967dc4764a50fde0f509797c7 P 262ca76d50824832b41b868a4b3c1560: Log is configured to *not* fsync() on all Append() calls
I20260621 05:14:22.781963 22680 tablet_bootstrap.cc:492] T 3d05b8c967dc4764a50fde0f509797c7 P c8590a7e27df4a6991e8c5be762b6ca0: No bootstrap required, opened a new log
I20260621 05:14:22.782020 22681 tablet_bootstrap.cc:492] T 3d05b8c967dc4764a50fde0f509797c7 P 262ca76d50824832b41b868a4b3c1560: No bootstrap required, opened a new log
I20260621 05:14:22.782033 22680 ts_tablet_manager.cc:1397] T 3d05b8c967dc4764a50fde0f509797c7 P c8590a7e27df4a6991e8c5be762b6ca0: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260621 05:14:22.782102 22681 ts_tablet_manager.cc:1397] T 3d05b8c967dc4764a50fde0f509797c7 P 262ca76d50824832b41b868a4b3c1560: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260621 05:14:22.784160 22680 raft_consensus.cc:348] T 3d05b8c967dc4764a50fde0f509797c7 P c8590a7e27df4a6991e8c5be762b6ca0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ddc0e22632743cb9a8a9f18596cb345" member_type: VOTER last_known_addr { host: "127.3.131.130" port: 46557 } } peers { permanent_uuid: "c8590a7e27df4a6991e8c5be762b6ca0" member_type: VOTER last_known_addr { host: "127.3.131.131" port: 41949 } } peers { permanent_uuid: "262ca76d50824832b41b868a4b3c1560" member_type: VOTER last_known_addr { host: "127.3.131.129" port: 33575 } }
I20260621 05:14:22.784328 22680 raft_consensus.cc:374] T 3d05b8c967dc4764a50fde0f509797c7 P c8590a7e27df4a6991e8c5be762b6ca0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 05:14:22.784371 22680 raft_consensus.cc:729] T 3d05b8c967dc4764a50fde0f509797c7 P c8590a7e27df4a6991e8c5be762b6ca0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c8590a7e27df4a6991e8c5be762b6ca0, State: Initialized, Role: FOLLOWER
I20260621 05:14:22.784487 22680 consensus_queue.cc:260] T 3d05b8c967dc4764a50fde0f509797c7 P c8590a7e27df4a6991e8c5be762b6ca0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ddc0e22632743cb9a8a9f18596cb345" member_type: VOTER last_known_addr { host: "127.3.131.130" port: 46557 } } peers { permanent_uuid: "c8590a7e27df4a6991e8c5be762b6ca0" member_type: VOTER last_known_addr { host: "127.3.131.131" port: 41949 } } peers { permanent_uuid: "262ca76d50824832b41b868a4b3c1560" member_type: VOTER last_known_addr { host: "127.3.131.129" port: 33575 } }
I20260621 05:14:22.784806 22666 heartbeater.cc:503] Master 127.3.131.190:44643 was elected leader, sending a full tablet report...
I20260621 05:14:22.785135 22685 tablet_bootstrap.cc:492] T 3d05b8c967dc4764a50fde0f509797c7 P 9ddc0e22632743cb9a8a9f18596cb345: Bootstrap starting.
I20260621 05:14:22.785085 22681 raft_consensus.cc:348] T 3d05b8c967dc4764a50fde0f509797c7 P 262ca76d50824832b41b868a4b3c1560 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ddc0e22632743cb9a8a9f18596cb345" member_type: VOTER last_known_addr { host: "127.3.131.130" port: 46557 } } peers { permanent_uuid: "c8590a7e27df4a6991e8c5be762b6ca0" member_type: VOTER last_known_addr { host: "127.3.131.131" port: 41949 } } peers { permanent_uuid: "262ca76d50824832b41b868a4b3c1560" member_type: VOTER last_known_addr { host: "127.3.131.129" port: 33575 } }
I20260621 05:14:22.785295 22681 raft_consensus.cc:374] T 3d05b8c967dc4764a50fde0f509797c7 P 262ca76d50824832b41b868a4b3c1560 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 05:14:22.785352 22681 raft_consensus.cc:729] T 3d05b8c967dc4764a50fde0f509797c7 P 262ca76d50824832b41b868a4b3c1560 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 262ca76d50824832b41b868a4b3c1560, State: Initialized, Role: FOLLOWER
I20260621 05:14:22.785476 22681 consensus_queue.cc:260] T 3d05b8c967dc4764a50fde0f509797c7 P 262ca76d50824832b41b868a4b3c1560 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ddc0e22632743cb9a8a9f18596cb345" member_type: VOTER last_known_addr { host: "127.3.131.130" port: 46557 } } peers { permanent_uuid: "c8590a7e27df4a6991e8c5be762b6ca0" member_type: VOTER last_known_addr { host: "127.3.131.131" port: 41949 } } peers { permanent_uuid: "262ca76d50824832b41b868a4b3c1560" member_type: VOTER last_known_addr { host: "127.3.131.129" port: 33575 } }
I20260621 05:14:22.785593 22680 ts_tablet_manager.cc:1428] T 3d05b8c967dc4764a50fde0f509797c7 P c8590a7e27df4a6991e8c5be762b6ca0: Time spent starting tablet: real 0.004s user 0.000s sys 0.003s
I20260621 05:14:22.785739 22681 ts_tablet_manager.cc:1428] T 3d05b8c967dc4764a50fde0f509797c7 P 262ca76d50824832b41b868a4b3c1560: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20260621 05:14:22.785872 22404 heartbeater.cc:503] Master 127.3.131.190:44643 was elected leader, sending a full tablet report...
I20260621 05:14:22.786453 22685 tablet_bootstrap.cc:654] T 3d05b8c967dc4764a50fde0f509797c7 P 9ddc0e22632743cb9a8a9f18596cb345: Neither blocks nor log segments found. Creating new log.
I20260621 05:14:22.786859 22685 log.cc:824] T 3d05b8c967dc4764a50fde0f509797c7 P 9ddc0e22632743cb9a8a9f18596cb345: Log is configured to *not* fsync() on all Append() calls
I20260621 05:14:22.787840 22685 tablet_bootstrap.cc:492] T 3d05b8c967dc4764a50fde0f509797c7 P 9ddc0e22632743cb9a8a9f18596cb345: No bootstrap required, opened a new log
I20260621 05:14:22.787925 22685 ts_tablet_manager.cc:1397] T 3d05b8c967dc4764a50fde0f509797c7 P 9ddc0e22632743cb9a8a9f18596cb345: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260621 05:14:22.789752 22685 raft_consensus.cc:348] T 3d05b8c967dc4764a50fde0f509797c7 P 9ddc0e22632743cb9a8a9f18596cb345 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ddc0e22632743cb9a8a9f18596cb345" member_type: VOTER last_known_addr { host: "127.3.131.130" port: 46557 } } peers { permanent_uuid: "c8590a7e27df4a6991e8c5be762b6ca0" member_type: VOTER last_known_addr { host: "127.3.131.131" port: 41949 } } peers { permanent_uuid: "262ca76d50824832b41b868a4b3c1560" member_type: VOTER last_known_addr { host: "127.3.131.129" port: 33575 } }
I20260621 05:14:22.789930 22685 raft_consensus.cc:374] T 3d05b8c967dc4764a50fde0f509797c7 P 9ddc0e22632743cb9a8a9f18596cb345 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 05:14:22.789978 22685 raft_consensus.cc:729] T 3d05b8c967dc4764a50fde0f509797c7 P 9ddc0e22632743cb9a8a9f18596cb345 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9ddc0e22632743cb9a8a9f18596cb345, State: Initialized, Role: FOLLOWER
I20260621 05:14:22.790107 22685 consensus_queue.cc:260] T 3d05b8c967dc4764a50fde0f509797c7 P 9ddc0e22632743cb9a8a9f18596cb345 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ddc0e22632743cb9a8a9f18596cb345" member_type: VOTER last_known_addr { host: "127.3.131.130" port: 46557 } } peers { permanent_uuid: "c8590a7e27df4a6991e8c5be762b6ca0" member_type: VOTER last_known_addr { host: "127.3.131.131" port: 41949 } } peers { permanent_uuid: "262ca76d50824832b41b868a4b3c1560" member_type: VOTER last_known_addr { host: "127.3.131.129" port: 33575 } }
I20260621 05:14:22.790380 22685 ts_tablet_manager.cc:1428] T 3d05b8c967dc4764a50fde0f509797c7 P 9ddc0e22632743cb9a8a9f18596cb345: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260621 05:14:22.790450 22535 heartbeater.cc:503] Master 127.3.131.190:44643 was elected leader, sending a full tablet report...
I20260621 05:14:22.804593 22688 raft_consensus.cc:482] T 3d05b8c967dc4764a50fde0f509797c7 P 9ddc0e22632743cb9a8a9f18596cb345 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260621 05:14:22.804823 22688 raft_consensus.cc:504] T 3d05b8c967dc4764a50fde0f509797c7 P 9ddc0e22632743cb9a8a9f18596cb345 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ddc0e22632743cb9a8a9f18596cb345" member_type: VOTER last_known_addr { host: "127.3.131.130" port: 46557 } } peers { permanent_uuid: "c8590a7e27df4a6991e8c5be762b6ca0" member_type: VOTER last_known_addr { host: "127.3.131.131" port: 41949 } } peers { permanent_uuid: "262ca76d50824832b41b868a4b3c1560" member_type: VOTER last_known_addr { host: "127.3.131.129" port: 33575 } }
I20260621 05:14:22.805214 22688 leader_election.cc:288] T 3d05b8c967dc4764a50fde0f509797c7 P 9ddc0e22632743cb9a8a9f18596cb345 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c8590a7e27df4a6991e8c5be762b6ca0 (127.3.131.131:41949), 262ca76d50824832b41b868a4b3c1560 (127.3.131.129:33575)
I20260621 05:14:22.809572 22678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:14:22.805627 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.131.130:49529 (local address 127.3.131.131:41949)
0621 05:14:22.805671 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0621 05:14:22.805673 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:14:22.805692 (+ 19us) server_negotiation.cc:407] Connection header received
0621 05:14:22.805760 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:14:22.805761 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:14:22.805770 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:14:22.805830 (+ 60us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 05:14:22.806043 (+ 213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:14:22.806833 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:14:22.808580 (+ 1747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:14:22.809159 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:14:22.809243 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:14:22.809251 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0621 05:14:22.809449 (+ 198us) server_negotiation.cc:299] Negotiation successful
0621 05:14:22.809495 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260621 05:14:22.809572 22679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:14:22.805584 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.131.130:47407 (local address 127.3.131.129:33575)
0621 05:14:22.805650 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0621 05:14:22.805652 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:14:22.805666 (+ 14us) server_negotiation.cc:407] Connection header received
0621 05:14:22.805709 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:14:22.805710 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:14:22.805723 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:14:22.805805 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 05:14:22.806043 (+ 238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:14:22.806810 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:14:22.808581 (+ 1771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:14:22.809167 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:14:22.809243 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:14:22.809251 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0621 05:14:22.809449 (+ 198us) server_negotiation.cc:299] Negotiation successful
0621 05:14:22.809495 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260621 05:14:22.809597 22689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:14:22.805449 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.131.131:41949 (local address 127.3.131.130:49529)
0621 05:14:22.805646 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0621 05:14:22.805655 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0621 05:14:22.805695 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:14:22.805837 (+ 142us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:14:22.805838 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:14:22.805839 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 05:14:22.805937 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:14:22.805938 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:14:22.806937 (+ 999us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:14:22.806938 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:14:22.808473 (+ 1535us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:14:22.808475 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:14:22.809261 (+ 786us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:14:22.809262 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:14:22.809349 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:14:22.809356 (+ 7us) client_negotiation.cc:769] Sending connection context
0621 05:14:22.809433 (+ 77us) client_negotiation.cc:240] Negotiation successful
0621 05:14:22.809483 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":183,"mutex_wait_us":260,"thread_start_us":98,"threads_started":1}
I20260621 05:14:22.809613 22537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:14:22.805449 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.131.129:33575 (local address 127.3.131.130:47407)
0621 05:14:22.805540 (+ 91us) negotiation.cc:107] Waiting for socket to connect
0621 05:14:22.805552 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0621 05:14:22.805644 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:14:22.805832 (+ 188us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:14:22.805833 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:14:22.805835 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 05:14:22.805932 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:14:22.805934 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:14:22.806937 (+ 1003us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:14:22.806938 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:14:22.808467 (+ 1529us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:14:22.808471 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:14:22.809261 (+ 790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:14:22.809262 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:14:22.809349 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:14:22.809356 (+ 7us) client_negotiation.cc:769] Sending connection context
0621 05:14:22.809435 (+ 79us) client_negotiation.cc:240] Negotiation successful
0621 05:14:22.809483 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":69,"mutex_wait_us":80}
I20260621 05:14:22.810017 22620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d05b8c967dc4764a50fde0f509797c7" candidate_uuid: "9ddc0e22632743cb9a8a9f18596cb345" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c8590a7e27df4a6991e8c5be762b6ca0" is_pre_election: true
I20260621 05:14:22.810220 22620 raft_consensus.cc:2454] T 3d05b8c967dc4764a50fde0f509797c7 P c8590a7e27df4a6991e8c5be762b6ca0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9ddc0e22632743cb9a8a9f18596cb345 in term 0.
I20260621 05:14:22.810449 22358 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d05b8c967dc4764a50fde0f509797c7" candidate_uuid: "9ddc0e22632743cb9a8a9f18596cb345" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "262ca76d50824832b41b868a4b3c1560" is_pre_election: true
I20260621 05:14:22.810595 22358 raft_consensus.cc:2454] T 3d05b8c967dc4764a50fde0f509797c7 P 262ca76d50824832b41b868a4b3c1560 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9ddc0e22632743cb9a8a9f18596cb345 in term 0.
I20260621 05:14:22.810956 22424 leader_election.cc:302] T 3d05b8c967dc4764a50fde0f509797c7 P 9ddc0e22632743cb9a8a9f18596cb345 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9ddc0e22632743cb9a8a9f18596cb345, c8590a7e27df4a6991e8c5be762b6ca0; no voters:
I20260621 05:14:22.811117 22688 raft_consensus.cc:2781] T 3d05b8c967dc4764a50fde0f509797c7 P 9ddc0e22632743cb9a8a9f18596cb345 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260621 05:14:22.811203 22688 raft_consensus.cc:482] T 3d05b8c967dc4764a50fde0f509797c7 P 9ddc0e22632743cb9a8a9f18596cb345 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260621 05:14:22.811235 22688 raft_consensus.cc:3037] T 3d05b8c967dc4764a50fde0f509797c7 P 9ddc0e22632743cb9a8a9f18596cb345 [term 0 FOLLOWER]: Advancing to term 1
I20260621 05:14:22.811959 22688 raft_consensus.cc:504] T 3d05b8c967dc4764a50fde0f509797c7 P 9ddc0e22632743cb9a8a9f18596cb345 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ddc0e22632743cb9a8a9f18596cb345" member_type: VOTER last_known_addr { host: "127.3.131.130" port: 46557 } } peers { permanent_uuid: "c8590a7e27df4a6991e8c5be762b6ca0" member_type: VOTER last_known_addr { host: "127.3.131.131" port: 41949 } } peers { permanent_uuid: "262ca76d50824832b41b868a4b3c1560" member_type: VOTER last_known_addr { host: "127.3.131.129" port: 33575 } }
I20260621 05:14:22.812165 22688 leader_election.cc:288] T 3d05b8c967dc4764a50fde0f509797c7 P 9ddc0e22632743cb9a8a9f18596cb345 [CANDIDATE]: Term 1 election: Requested vote from peers c8590a7e27df4a6991e8c5be762b6ca0 (127.3.131.131:41949), 262ca76d50824832b41b868a4b3c1560 (127.3.131.129:33575)
I20260621 05:14:22.812261 22620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d05b8c967dc4764a50fde0f509797c7" candidate_uuid: "9ddc0e22632743cb9a8a9f18596cb345" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c8590a7e27df4a6991e8c5be762b6ca0"
I20260621 05:14:22.812279 22358 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d05b8c967dc4764a50fde0f509797c7" candidate_uuid: "9ddc0e22632743cb9a8a9f18596cb345" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "262ca76d50824832b41b868a4b3c1560"
I20260621 05:14:22.812347 22620 raft_consensus.cc:3037] T 3d05b8c967dc4764a50fde0f509797c7 P c8590a7e27df4a6991e8c5be762b6ca0 [term 0 FOLLOWER]: Advancing to term 1
I20260621 05:14:22.812351 22358 raft_consensus.cc:3037] T 3d05b8c967dc4764a50fde0f509797c7 P 262ca76d50824832b41b868a4b3c1560 [term 0 FOLLOWER]: Advancing to term 1
I20260621 05:14:22.813031 22358 raft_consensus.cc:2454] T 3d05b8c967dc4764a50fde0f509797c7 P 262ca76d50824832b41b868a4b3c1560 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9ddc0e22632743cb9a8a9f18596cb345 in term 1.
I20260621 05:14:22.813117 22620 raft_consensus.cc:2454] T 3d05b8c967dc4764a50fde0f509797c7 P c8590a7e27df4a6991e8c5be762b6ca0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9ddc0e22632743cb9a8a9f18596cb345 in term 1.
I20260621 05:14:22.813222 22422 leader_election.cc:302] T 3d05b8c967dc4764a50fde0f509797c7 P 9ddc0e22632743cb9a8a9f18596cb345 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 262ca76d50824832b41b868a4b3c1560, 9ddc0e22632743cb9a8a9f18596cb345; no voters:
I20260621 05:14:22.813309 22688 raft_consensus.cc:2781] T 3d05b8c967dc4764a50fde0f509797c7 P 9ddc0e22632743cb9a8a9f18596cb345 [term 1 FOLLOWER]: Leader election won for term 1
I20260621 05:14:22.813454 22688 raft_consensus.cc:686] T 3d05b8c967dc4764a50fde0f509797c7 P 9ddc0e22632743cb9a8a9f18596cb345 [term 1 LEADER]: Becoming Leader. State: Replica: 9ddc0e22632743cb9a8a9f18596cb345, State: Running, Role: LEADER
I20260621 05:14:22.813587 22688 consensus_queue.cc:237] T 3d05b8c967dc4764a50fde0f509797c7 P 9ddc0e22632743cb9a8a9f18596cb345 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ddc0e22632743cb9a8a9f18596cb345" member_type: VOTER last_known_addr { host: "127.3.131.130" port: 46557 } } peers { permanent_uuid: "c8590a7e27df4a6991e8c5be762b6ca0" member_type: VOTER last_known_addr { host: "127.3.131.131" port: 41949 } } peers { permanent_uuid: "262ca76d50824832b41b868a4b3c1560" member_type: VOTER last_known_addr { host: "127.3.131.129" port: 33575 } }
I20260621 05:14:22.814429 22216 catalog_manager.cc:5495] T 3d05b8c967dc4764a50fde0f509797c7 P 9ddc0e22632743cb9a8a9f18596cb345 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9ddc0e22632743cb9a8a9f18596cb345 (127.3.131.130). New cstate: current_term: 1 leader_uuid: "9ddc0e22632743cb9a8a9f18596cb345" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ddc0e22632743cb9a8a9f18596cb345" member_type: VOTER last_known_addr { host: "127.3.131.130" port: 46557 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c8590a7e27df4a6991e8c5be762b6ca0" member_type: VOTER last_known_addr { host: "127.3.131.131" port: 41949 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "262ca76d50824832b41b868a4b3c1560" member_type: VOTER last_known_addr { host: "127.3.131.129" port: 33575 } health_report { overall_health: UNKNOWN } } }
I20260621 05:14:22.822197 22677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:14:22.819734 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47444 (local address 127.3.131.130:46557)
0621 05:14:22.819795 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0621 05:14:22.819797 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:14:22.819805 (+ 8us) server_negotiation.cc:407] Connection header received
0621 05:14:22.819838 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:14:22.819839 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:14:22.819856 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:14:22.819919 (+ 63us) server_negotiation.cc:226] Negotiated authn=TOKEN
0621 05:14:22.820064 (+ 145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:14:22.820535 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:14:22.821357 (+ 822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:14:22.821557 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:14:22.821666 (+ 109us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:14:22.821800 (+ 134us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0621 05:14:22.821885 (+ 85us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0621 05:14:22.821960 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0621 05:14:22.822050 (+ 90us) server_negotiation.cc:299] Negotiation successful
0621 05:14:22.822059 (+ 9us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260621 05:14:22.824397 22358 raft_consensus.cc:1261] T 3d05b8c967dc4764a50fde0f509797c7 P 262ca76d50824832b41b868a4b3c1560 [term 1 FOLLOWER]: Refusing update from remote peer 9ddc0e22632743cb9a8a9f18596cb345: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 05:14:22.824393 22620 raft_consensus.cc:1261] T 3d05b8c967dc4764a50fde0f509797c7 P c8590a7e27df4a6991e8c5be762b6ca0 [term 1 FOLLOWER]: Refusing update from remote peer 9ddc0e22632743cb9a8a9f18596cb345: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 05:14:22.824780 22688 consensus_queue.cc:1035] T 3d05b8c967dc4764a50fde0f509797c7 P 9ddc0e22632743cb9a8a9f18596cb345 [LEADER]: Connected to new peer: Peer: permanent_uuid: "262ca76d50824832b41b868a4b3c1560" member_type: VOTER last_known_addr { host: "127.3.131.129" port: 33575 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 05:14:22.824939 22690 consensus_queue.cc:1035] T 3d05b8c967dc4764a50fde0f509797c7 P 9ddc0e22632743cb9a8a9f18596cb345 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8590a7e27df4a6991e8c5be762b6ca0" member_type: VOTER last_known_addr { host: "127.3.131.131" port: 41949 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 05:14:22.827626 22695 mvcc.cc:205] Tried to move back new op lower bound from 7299149262124654592 to 7299149262085115904. Current Snapshot: MvccSnapshot[applied={T|T < 7299149262124654592}]
I20260621 05:14:22.829897 22216 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41914:
table { table_name: "test-table" } modify_external_catalogs: true
I20260621 05:14:22.830008 22216 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41914:
table { table_name: "test-table" } modify_external_catalogs: true
I20260621 05:14:22.830818 22216 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c107cc5843fa43dfaa7df9a785c15a03: Sending DeleteTablet for 3 replicas of tablet 3d05b8c967dc4764a50fde0f509797c7
I20260621 05:14:22.831112 22469 tablet_service.cc:1514] Processing DeleteTablet for tablet 3d05b8c967dc4764a50fde0f509797c7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 05:14:22 UTC) from {username='slave'} at 127.0.0.1:47442
I20260621 05:14:22.831230 22338 tablet_service.cc:1514] Processing DeleteTablet for tablet 3d05b8c967dc4764a50fde0f509797c7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 05:14:22 UTC) from {username='slave'} at 127.0.0.1:33174
I20260621 05:14:22.831391 22703 tablet_replica.cc:330] T 3d05b8c967dc4764a50fde0f509797c7 P 9ddc0e22632743cb9a8a9f18596cb345: stopping tablet replica
I20260621 05:14:22.831373 22600 tablet_service.cc:1514] Processing DeleteTablet for tablet 3d05b8c967dc4764a50fde0f509797c7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 05:14:22 UTC) from {username='slave'} at 127.0.0.1:42330
I20260621 05:14:22.831490 22703 raft_consensus.cc:2229] T 3d05b8c967dc4764a50fde0f509797c7 P 9ddc0e22632743cb9a8a9f18596cb345 [term 1 LEADER]: Raft consensus shutting down.
I20260621 05:14:22.831583 22703 raft_consensus.cc:2258] T 3d05b8c967dc4764a50fde0f509797c7 P 9ddc0e22632743cb9a8a9f18596cb345 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 05:14:22.831601 22704 tablet_replica.cc:330] T 3d05b8c967dc4764a50fde0f509797c7 P 262ca76d50824832b41b868a4b3c1560: stopping tablet replica
I20260621 05:14:22.831791 3598 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWZOx2_/build/release/bin/kudu with pid 22276
I20260621 05:14:22.831794 22704 raft_consensus.cc:2229] T 3d05b8c967dc4764a50fde0f509797c7 P 262ca76d50824832b41b868a4b3c1560 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260621 05:14:22.831899 22704 raft_consensus.cc:2258] T 3d05b8c967dc4764a50fde0f509797c7 P 262ca76d50824832b41b868a4b3c1560 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 05:14:22.832377 22705 tablet_replica.cc:330] T 3d05b8c967dc4764a50fde0f509797c7 P c8590a7e27df4a6991e8c5be762b6ca0: stopping tablet replica
I20260621 05:14:22.832484 22705 raft_consensus.cc:2229] T 3d05b8c967dc4764a50fde0f509797c7 P c8590a7e27df4a6991e8c5be762b6ca0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260621 05:14:22.832592 22705 raft_consensus.cc:2258] T 3d05b8c967dc4764a50fde0f509797c7 P c8590a7e27df4a6991e8c5be762b6ca0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 05:14:22.833925 22703 ts_tablet_manager.cc:1905] T 3d05b8c967dc4764a50fde0f509797c7 P 9ddc0e22632743cb9a8a9f18596cb345: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 05:14:22.836027 22703 ts_tablet_manager.cc:1918] T 3d05b8c967dc4764a50fde0f509797c7 P 9ddc0e22632743cb9a8a9f18596cb345: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260621 05:14:22.836115 22703 log.cc:1196] T 3d05b8c967dc4764a50fde0f509797c7 P 9ddc0e22632743cb9a8a9f18596cb345: Deleting WAL directory at /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782018616648578-3598-0/minicluster-data/ts-1/wal/wals/3d05b8c967dc4764a50fde0f509797c7
I20260621 05:14:22.836366 22703 ts_tablet_manager.cc:1939] T 3d05b8c967dc4764a50fde0f509797c7 P 9ddc0e22632743cb9a8a9f18596cb345: Deleting consensus metadata
I20260621 05:14:22.836877 22203 catalog_manager.cc:4841] TS 9ddc0e22632743cb9a8a9f18596cb345 (127.3.131.130:46557): tablet 3d05b8c967dc4764a50fde0f509797c7 (table test-table [id=459918e4a5424cf3bec26e57ee0fa7ce]) successfully deleted
W20260621 05:14:22.838521 22200 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.3.131.129:33575 (error 108)
I20260621 05:14:22.839011 22676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:14:22.838884 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.131.129:33575 (local address 127.0.0.1:33178)
0621 05:14:22.838936 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0621 05:14:22.838970 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.131.129:33575: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
I20260621 05:14:22.839116 3598 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWZOx2_/build/release/bin/kudu with pid 22407
I20260621 05:14:22.839118 22705 ts_tablet_manager.cc:1905] T 3d05b8c967dc4764a50fde0f509797c7 P c8590a7e27df4a6991e8c5be762b6ca0: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260621 05:14:22.839321 22200 catalog_manager.cc:4584] TS 262ca76d50824832b41b868a4b3c1560 (127.3.131.129:33575): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 3d05b8c967dc4764a50fde0f509797c7: Network error: Client connection negotiation failed: client connection to 127.3.131.129:33575: connect: Connection refused (error 111)
I20260621 05:14:22.841358 22705 ts_tablet_manager.cc:1918] T 3d05b8c967dc4764a50fde0f509797c7 P c8590a7e27df4a6991e8c5be762b6ca0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260621 05:14:22.841452 22705 log.cc:1196] T 3d05b8c967dc4764a50fde0f509797c7 P c8590a7e27df4a6991e8c5be762b6ca0: Deleting WAL directory at /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782018616648578-3598-0/minicluster-data/ts-2/wal/wals/3d05b8c967dc4764a50fde0f509797c7
I20260621 05:14:22.841744 22705 ts_tablet_manager.cc:1939] T 3d05b8c967dc4764a50fde0f509797c7 P c8590a7e27df4a6991e8c5be762b6ca0: Deleting consensus metadata
I20260621 05:14:22.842245 22202 catalog_manager.cc:4841] TS c8590a7e27df4a6991e8c5be762b6ca0 (127.3.131.131:41949): tablet 3d05b8c967dc4764a50fde0f509797c7 (table test-table [id=459918e4a5424cf3bec26e57ee0fa7ce]) successfully deleted
I20260621 05:14:22.844811 3598 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWZOx2_/build/release/bin/kudu with pid 22538
I20260621 05:14:22.849485 3598 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWZOx2_/build/release/bin/kudu with pid 22185
2026-06-21T05:14:22Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (707 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782018616648578-3598-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 21 05:14:22 dist-test-slave-vrq1 krb5kdc[22709](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 21 05:14:22 dist-test-slave-vrq1 krb5kdc[22709](info): set up 2 sockets
Jun 21 05:14:22 dist-test-slave-vrq1 krb5kdc[22709](info): commencing operation
krb5kdc: starting...
W20260621 05:14:23.931733 3598 mini_kdc.cc:121] Time spent starting KDC: real 1.063s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782018616648578-3598-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782018616648578-3598-0/krb5kdc/test-user.keytab.
Jun 21 05:14:24 dist-test-slave-vrq1 krb5kdc[22709](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782018864, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-21T05:14:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-21T05:14:24Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.131.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.131.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.131.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782018616648578-3598-0/krb5kdc/kudu_127.3.131.190.keytab.
Entry for principal kudu/127.3.131.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782018616648578-3598-0/krb5kdc/kudu_127.3.131.190.keytab.
I20260621 05:14:24.071378 3598 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWZOx2_/build/release/bin/kudu
/tmp/dist-test-taskWZOx2_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782018616648578-3598-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782018616648578-3598-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782018616648578-3598-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782018616648578-3598-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.131.190:44125
--webserver_interface=127.3.131.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782018616648578-3598-0/krb5kdc/kudu_127.3.131.190.keytab
--principal=kudu/127.3.131.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.131.148:40953
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.131.190:44125
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782018616648578-3598-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782018616648578-3598-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782018616648578-3598-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 05:14:24.155884 22722 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 05:14:24.156140 22722 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 05:14:24.156185 22722 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 05:14:24.157382 22722 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 05:14:24.157454 22722 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 05:14:24.157480 22722 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 05:14:24.157508 22722 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 05:14:24.157541 22722 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 05:14:24.159106 22722 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.131.148:40953
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782018616648578-3598-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782018616648578-3598-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.131.190:44125
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782018616648578-3598-0/krb5kdc/kudu_127.3.131.190.keytab
--principal=kudu/127.3.131.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.131.190:44125
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782018616648578-3598-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.131.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22722
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782018616648578-3598-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 21 Jun 2026 04:16:16 UTC on ccycloud.quasar-xycaha.root.comops.site
build id 4457881
I20260621 05:14:24.159432 22722 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 05:14:24.159679 22722 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 05:14:24.164855 22730 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:14:24.164862 22728 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:14:24.164920 22727 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:14:24.165336 22722 server_base.cc:1034] running on GCE node
I20260621 05:14:24.165583 22722 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 05:14:24.165912 22722 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 05:14:24.167122 22722 hybrid_clock.cc:648] HybridClock initialized: now 1782018864167109 us; error 39 us; skew 500 ppm
Jun 21 05:14:24 dist-test-slave-vrq1 krb5kdc[22709](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782018864, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.131.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 05:14:24.169986 22722 init.cc:377] Logged in from keytab as kudu/127.3.131.190@KRBTEST.COM (short username kudu)
I20260621 05:14:24.170459 22722 webserver.cc:460] Webserver started at http://127.3.131.190:34513/ using document root <none> and password file <none>
I20260621 05:14:24.170696 22722 fs_manager.cc:359] Metadata directory not provided
I20260621 05:14:24.170764 22722 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 05:14:24.170912 22722 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 05:14:24.172088 22722 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782018616648578-3598-0/minicluster-data/master-0/data/instance:
uuid: "0c7f8bbeafb44227b9a2d125fc4ad530"
format_stamp: "Formatted at 2026-06-21 05:14:24 on dist-test-slave-vrq1"
server_key: "f88e6f4242bc5f3693f6cfe5ce452d39"
server_key_iv: "94c9c1ed6bffdf661cdc1734b3bbbe5d"
server_key_version: "encryptionkey@0"
I20260621 05:14:24.172466 22722 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782018616648578-3598-0/minicluster-data/master-0/wal/instance:
uuid: "0c7f8bbeafb44227b9a2d125fc4ad530"
format_stamp: "Formatted at 2026-06-21 05:14:24 on dist-test-slave-vrq1"
server_key: "f88e6f4242bc5f3693f6cfe5ce452d39"
server_key_iv: "94c9c1ed6bffdf661cdc1734b3bbbe5d"
server_key_version: "encryptionkey@0"
I20260621 05:14:24.173945 22722 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260621 05:14:24.174736 22737 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 05:14:24.174944 22722 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260621 05:14:24.175050 22722 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782018616648578-3598-0/minicluster-data/master-0/data,/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782018616648578-3598-0/minicluster-data/master-0/wal
uuid: "0c7f8bbeafb44227b9a2d125fc4ad530"
format_stamp: "Formatted at 2026-06-21 05:14:24 on dist-test-slave-vrq1"
server_key: "f88e6f4242bc5f3693f6cfe5ce452d39"
server_key_iv: "94c9c1ed6bffdf661cdc1734b3bbbe5d"
server_key_version: "encryptionkey@0"
I20260621 05:14:24.175125 22722 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782018616648578-3598-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782018616648578-3598-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782018616648578-3598-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 05:14:24.192823 22722 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 05:14:24.195443 22722 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 05:14:24.195608 22722 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 05:14:24.199795 22722 rpc_server.cc:307] RPC server started. Bound to: 127.3.131.190:44125
I20260621 05:14:24.199837 22789 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.131.190:44125 every 8 connection(s)
I20260621 05:14:24.200210 22722 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782018616648578-3598-0/minicluster-data/master-0/data/info.pb
I20260621 05:14:24.201140 22790 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 05:14:24.203552 22790 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0c7f8bbeafb44227b9a2d125fc4ad530: Bootstrap starting.
I20260621 05:14:24.204159 22790 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0c7f8bbeafb44227b9a2d125fc4ad530: Neither blocks nor log segments found. Creating new log.
I20260621 05:14:24.204491 22790 log.cc:824] T 00000000000000000000000000000000 P 0c7f8bbeafb44227b9a2d125fc4ad530: Log is configured to *not* fsync() on all Append() calls
I20260621 05:14:24.205300 22790 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0c7f8bbeafb44227b9a2d125fc4ad530: No bootstrap required, opened a new log
I20260621 05:14:24.206979 22790 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0c7f8bbeafb44227b9a2d125fc4ad530 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c7f8bbeafb44227b9a2d125fc4ad530" member_type: VOTER last_known_addr { host: "127.3.131.190" port: 44125 } }
I20260621 05:14:24.207135 22790 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0c7f8bbeafb44227b9a2d125fc4ad530 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 05:14:24.207154 3598 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWZOx2_/build/release/bin/kudu as pid 22722
I20260621 05:14:24.207159 22790 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0c7f8bbeafb44227b9a2d125fc4ad530 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0c7f8bbeafb44227b9a2d125fc4ad530, State: Initialized, Role: FOLLOWER
I20260621 05:14:24.207300 3598 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782018616648578-3598-0/minicluster-data/master-0/wal/instance
I20260621 05:14:24.207373 22790 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0c7f8bbeafb44227b9a2d125fc4ad530 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c7f8bbeafb44227b9a2d125fc4ad530" member_type: VOTER last_known_addr { host: "127.3.131.190" port: 44125 } }
I20260621 05:14:24.207468 22790 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0c7f8bbeafb44227b9a2d125fc4ad530 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260621 05:14:24.207507 22790 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0c7f8bbeafb44227b9a2d125fc4ad530 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260621 05:14:24.207512 3598 external_mini_cluster.cc:1428] Setting key d2a445686896751cb9dce5cfe46f0713
I20260621 05:14:24.207557 22790 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0c7f8bbeafb44227b9a2d125fc4ad530 [term 0 FOLLOWER]: Advancing to term 1
I20260621 05:14:24.208315 22790 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0c7f8bbeafb44227b9a2d125fc4ad530 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c7f8bbeafb44227b9a2d125fc4ad530" member_type: VOTER last_known_addr { host: "127.3.131.190" port: 44125 } }
I20260621 05:14:24.208446 22790 leader_election.cc:302] T 00000000000000000000000000000000 P 0c7f8bbeafb44227b9a2d125fc4ad530 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0c7f8bbeafb44227b9a2d125fc4ad530; no voters:
I20260621 05:14:24.208626 22790 leader_election.cc:288] T 00000000000000000000000000000000 P 0c7f8bbeafb44227b9a2d125fc4ad530 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260621 05:14:24.208647 22795 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0c7f8bbeafb44227b9a2d125fc4ad530 [term 1 FOLLOWER]: Leader election won for term 1
I20260621 05:14:24.208801 22795 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0c7f8bbeafb44227b9a2d125fc4ad530 [term 1 LEADER]: Becoming Leader. State: Replica: 0c7f8bbeafb44227b9a2d125fc4ad530, State: Running, Role: LEADER
I20260621 05:14:24.208885 22790 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0c7f8bbeafb44227b9a2d125fc4ad530 [sys.catalog]: configured and running, proceeding with master startup.
I20260621 05:14:24.208899 22795 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0c7f8bbeafb44227b9a2d125fc4ad530 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c7f8bbeafb44227b9a2d125fc4ad530" member_type: VOTER last_known_addr { host: "127.3.131.190" port: 44125 } }
I20260621 05:14:24.209236 22797 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0c7f8bbeafb44227b9a2d125fc4ad530 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0c7f8bbeafb44227b9a2d125fc4ad530" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c7f8bbeafb44227b9a2d125fc4ad530" member_type: VOTER last_known_addr { host: "127.3.131.190" port: 44125 } } }
I20260621 05:14:24.209327 22797 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0c7f8bbeafb44227b9a2d125fc4ad530 [sys.catalog]: This master's current role is: LEADER
I20260621 05:14:24.209240 22796 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0c7f8bbeafb44227b9a2d125fc4ad530 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0c7f8bbeafb44227b9a2d125fc4ad530. Latest consensus state: current_term: 1 leader_uuid: "0c7f8bbeafb44227b9a2d125fc4ad530" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c7f8bbeafb44227b9a2d125fc4ad530" member_type: VOTER last_known_addr { host: "127.3.131.190" port: 44125 } } }
I20260621 05:14:24.209537 22796 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0c7f8bbeafb44227b9a2d125fc4ad530 [sys.catalog]: This master's current role is: LEADER
I20260621 05:14:24.209676 22800 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260621 05:14:24.210474 22800 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Jun 21 05:14:24 dist-test-slave-vrq1 krb5kdc[22709](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782018864, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.131.190@KRBTEST.COM
I20260621 05:14:24.215526 22800 catalog_manager.cc:1347] Generated new cluster ID: 1ddd88043dd04432b645fdce38f78eed
I20260621 05:14:24.215608 22800 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260621 05:14:24.220798 22794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:14:24.207989 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37902 (local address 127.3.131.190:44125)
0621 05:14:24.208627 (+ 638us) server_negotiation.cc:206] Beginning negotiation
0621 05:14:24.208637 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0621 05:14:24.208674 (+ 37us) server_negotiation.cc:407] Connection header received
0621 05:14:24.208803 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:14:24.208810 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:14:24.208867 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:14:24.209942 (+ 1075us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:14:24.210071 (+ 129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:14:24.211529 (+ 1458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:14:24.212360 (+ 831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:14:24.212991 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:14:24.213107 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:14:24.215857 (+ 2750us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:14:24.215861 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:14:24.215861 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:14:24.215874 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:14:24.218820 (+ 2946us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:14:24.219393 (+ 573us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:14:24.219396 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:14:24.219396 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:14:24.219520 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:14:24.219768 (+ 248us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:14:24.219769 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:14:24.219769 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:14:24.220077 (+ 308us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:14:24.220218 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0621 05:14:24.220421 (+ 203us) server_negotiation.cc:299] Negotiation successful
0621 05:14:24.220563 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":574,"thread_start_us":540,"threads_started":1}
I20260621 05:14:24.228052 22800 catalog_manager.cc:1370] Generated new certificate authority record
I20260621 05:14:24.228601 22800 catalog_manager.cc:1504] Loading token signing keys...
I20260621 05:14:24.240115 22800 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0c7f8bbeafb44227b9a2d125fc4ad530: Generated new TSK 0
I20260621 05:14:24.240396 22800 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.3.131.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.131.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.131.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782018616648578-3598-0/krb5kdc/kudu_127.3.131.129.keytab.
Entry for principal kudu/127.3.131.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782018616648578-3598-0/krb5kdc/kudu_127.3.131.129.keytab.
I20260621 05:14:24.274333 3598 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWZOx2_/build/release/bin/kudu
/tmp/dist-test-taskWZOx2_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782018616648578-3598-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782018616648578-3598-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782018616648578-3598-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782018616648578-3598-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.131.129:0
--local_ip_for_outbound_sockets=127.3.131.129
--webserver_interface=127.3.131.129
--webserver_port=0
--tserver_master_addrs=127.3.131.190:44125
--keytab_file=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782018616648578-3598-0/krb5kdc/kudu_127.3.131.129.keytab
--principal=kudu/127.3.131.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.131.148:40953
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782018616648578-3598-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782018616648578-3598-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782018616648578-3598-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 05:14:24.379006 22816 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 05:14:24.379321 22816 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 05:14:24.379350 22816 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 05:14:24.380538 22816 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 05:14:24.380587 22816 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 05:14:24.380614 22816 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.131.129
I20260621 05:14:24.382647 22816 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.131.148:40953
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782018616648578-3598-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782018616648578-3598-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782018616648578-3598-0/krb5kdc/kudu_127.3.131.129.keytab
--principal=kudu/127.3.131.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.131.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782018616648578-3598-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.131.129
--webserver_port=0
--tserver_master_addrs=127.3.131.190:44125
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22816
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.131.129
--log_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782018616648578-3598-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 21 Jun 2026 04:16:16 UTC on ccycloud.quasar-xycaha.root.comops.site
build id 4457881
I20260621 05:14:24.382910 22816 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 05:14:24.383226 22816 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 05:14:24.389075 22821 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:14:24.389168 22822 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:14:24.389143 22824 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:14:24.389319 22816 server_base.cc:1034] running on GCE node
I20260621 05:14:24.389532 22816 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 05:14:24.389808 22816 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 05:14:24.390996 22816 hybrid_clock.cc:648] HybridClock initialized: now 1782018864390967 us; error 40 us; skew 500 ppm
Jun 21 05:14:24 dist-test-slave-vrq1 krb5kdc[22709](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782018864, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.131.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 05:14:24.393878 22816 init.cc:377] Logged in from keytab as kudu/127.3.131.129@KRBTEST.COM (short username kudu)
I20260621 05:14:24.394471 22816 webserver.cc:460] Webserver started at http://127.3.131.129:39719/ using document root <none> and password file <none>
I20260621 05:14:24.394717 22816 fs_manager.cc:359] Metadata directory not provided
I20260621 05:14:24.394786 22816 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 05:14:24.394929 22816 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 05:14:24.396129 22816 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782018616648578-3598-0/minicluster-data/ts-0/data/instance:
uuid: "fe661fee975b4b2587e1f23dcdbddf9b"
format_stamp: "Formatted at 2026-06-21 05:14:24 on dist-test-slave-vrq1"
server_key: "4afd87af843fd0f0f1f3b2f62fe6b11e"
server_key_iv: "6d8f4590c3fcf03828dacb330a9f3f3f"
server_key_version: "encryptionkey@0"
I20260621 05:14:24.396538 22816 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782018616648578-3598-0/minicluster-data/ts-0/wal/instance:
uuid: "fe661fee975b4b2587e1f23dcdbddf9b"
format_stamp: "Formatted at 2026-06-21 05:14:24 on dist-test-slave-vrq1"
server_key: "4afd87af843fd0f0f1f3b2f62fe6b11e"
server_key_iv: "6d8f4590c3fcf03828dacb330a9f3f3f"
server_key_version: "encryptionkey@0"
I20260621 05:14:24.398089 22816 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260621 05:14:24.399060 22831 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 05:14:24.399379 22816 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260621 05:14:24.399520 22816 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782018616648578-3598-0/minicluster-data/ts-0/data,/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782018616648578-3598-0/minicluster-data/ts-0/wal
uuid: "fe661fee975b4b2587e1f23dcdbddf9b"
format_stamp: "Formatted at 2026-06-21 05:14:24 on dist-test-slave-vrq1"
server_key: "4afd87af843fd0f0f1f3b2f62fe6b11e"
server_key_iv: "6d8f4590c3fcf03828dacb330a9f3f3f"
server_key_version: "encryptionkey@0"
I20260621 05:14:24.399607 22816 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782018616648578-3598-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782018616648578-3598-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782018616648578-3598-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 05:14:24.424086 22816 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 05:14:24.426431 22816 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 05:14:24.426606 22816 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 05:14:24.426904 22816 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 05:14:24.427244 22816 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 05:14:24.427296 22816 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 05:14:24.427330 22816 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 05:14:24.427358 22816 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 05:14:24.433239 22816 rpc_server.cc:307] RPC server started. Bound to: 127.3.131.129:39779
I20260621 05:14:24.433287 22944 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.131.129:39779 every 8 connection(s)
I20260621 05:14:24.433610 22816 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782018616648578-3598-0/minicluster-data/ts-0/data/info.pb
Jun 21 05:14:24 dist-test-slave-vrq1 krb5kdc[22709](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782018864, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.131.129@KRBTEST.COM for kudu/127.3.131.190@KRBTEST.COM
I20260621 05:14:24.440682 3598 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWZOx2_/build/release/bin/kudu as pid 22816
I20260621 05:14:24.440819 3598 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782018616648578-3598-0/minicluster-data/ts-0/wal/instance
I20260621 05:14:24.441048 3598 external_mini_cluster.cc:1428] Setting key 60d7ad85ae15fadadbd998dc05cc9b34
I20260621 05:14:24.443869 22794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:14:24.434356 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.131.129:43559 (local address 127.3.131.190:44125)
0621 05:14:24.434419 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0621 05:14:24.434421 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:14:24.434652 (+ 231us) server_negotiation.cc:407] Connection header received
0621 05:14:24.435276 (+ 624us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:14:24.435278 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:14:24.435293 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:14:24.435357 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:14:24.436506 (+ 1149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:14:24.437022 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:14:24.438068 (+ 1046us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:14:24.438288 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:14:24.438353 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:14:24.440785 (+ 2432us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:14:24.440787 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:14:24.440787 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:14:24.440795 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:14:24.442343 (+ 1548us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:14:24.442817 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:14:24.442819 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:14:24.442819 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:14:24.442877 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:14:24.443138 (+ 261us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:14:24.443139 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:14:24.443140 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:14:24.443316 (+ 176us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:14:24.443393 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0621 05:14:24.443673 (+ 280us) server_negotiation.cc:299] Negotiation successful
0621 05:14:24.443768 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260621 05:14:24.444097 22947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:14:24.434326 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.131.190:44125 (local address 127.3.131.129:43559)
0621 05:14:24.434499 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0621 05:14:24.434543 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0621 05:14:24.435038 (+ 495us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:14:24.435466 (+ 428us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:14:24.435470 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:14:24.436168 (+ 698us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:14:24.436392 (+ 224us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:14:24.436397 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:14:24.437129 (+ 732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:14:24.437130 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:14:24.437976 (+ 846us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:14:24.437978 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:14:24.438376 (+ 398us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:14:24.438377 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:14:24.438480 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:14:24.439045 (+ 565us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 05:14:24.439054 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:14:24.440625 (+ 1571us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 05:14:24.442429 (+ 1804us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:14:24.442437 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:14:24.442440 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:14:24.442725 (+ 285us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:14:24.442941 (+ 216us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:14:24.442942 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:14:24.442942 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:14:24.443062 (+ 120us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:14:24.443381 (+ 319us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 05:14:24.443389 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 05:14:24.443540 (+ 151us) client_negotiation.cc:769] Sending connection context
0621 05:14:24.443652 (+ 112us) client_negotiation.cc:240] Negotiation successful
0621 05:14:24.443847 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":85,"threads_started":1}
I20260621 05:14:24.444590 22945 heartbeater.cc:347] Connected to a master server at 127.3.131.190:44125
I20260621 05:14:24.444712 22945 heartbeater.cc:464] Registering TS with master...
I20260621 05:14:24.444921 22945 heartbeater.cc:511] Master 127.3.131.190:44125 requested a full tablet report, sending...
I20260621 05:14:24.445551 22754 ts_manager.cc:194] Registered new tserver with Master: fe661fee975b4b2587e1f23dcdbddf9b (127.3.131.129:39779)
I20260621 05:14:24.446211 22754 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.131.129@KRBTEST.COM'} at 127.3.131.129:43559
WARNING: no policy specified for kudu/127.3.131.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.131.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.131.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782018616648578-3598-0/krb5kdc/kudu_127.3.131.130.keytab.
Entry for principal kudu/127.3.131.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782018616648578-3598-0/krb5kdc/kudu_127.3.131.130.keytab.
I20260621 05:14:24.470809 3598 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWZOx2_/build/release/bin/kudu
/tmp/dist-test-taskWZOx2_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782018616648578-3598-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782018616648578-3598-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782018616648578-3598-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782018616648578-3598-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.131.130:0
--local_ip_for_outbound_sockets=127.3.131.130
--webserver_interface=127.3.131.130
--webserver_port=0
--tserver_master_addrs=127.3.131.190:44125
--keytab_file=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782018616648578-3598-0/krb5kdc/kudu_127.3.131.130.keytab
--principal=kudu/127.3.131.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.131.148:40953
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782018616648578-3598-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782018616648578-3598-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782018616648578-3598-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 05:14:24.556205 22950 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 05:14:24.556442 22950 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 05:14:24.556471 22950 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 05:14:24.557673 22950 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 05:14:24.557714 22950 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 05:14:24.557744 22950 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.131.130
I20260621 05:14:24.559206 22950 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.131.148:40953
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782018616648578-3598-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782018616648578-3598-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782018616648578-3598-0/krb5kdc/kudu_127.3.131.130.keytab
--principal=kudu/127.3.131.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.131.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782018616648578-3598-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.131.130
--webserver_port=0
--tserver_master_addrs=127.3.131.190:44125
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22950
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.131.130
--log_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782018616648578-3598-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 21 Jun 2026 04:16:16 UTC on ccycloud.quasar-xycaha.root.comops.site
build id 4457881
I20260621 05:14:24.559414 22950 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 05:14:24.559602 22950 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260621 05:14:24.565003 22950 server_base.cc:1034] running on GCE node
W20260621 05:14:24.564944 22958 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:14:24.564944 22955 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:14:24.564939 22956 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:14:24.565512 22950 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 05:14:24.565775 22950 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 05:14:24.566957 22950 hybrid_clock.cc:648] HybridClock initialized: now 1782018864566933 us; error 38 us; skew 500 ppm
Jun 21 05:14:24 dist-test-slave-vrq1 krb5kdc[22709](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782018864, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.131.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 05:14:24.569710 22950 init.cc:377] Logged in from keytab as kudu/127.3.131.130@KRBTEST.COM (short username kudu)
I20260621 05:14:24.570194 22950 webserver.cc:460] Webserver started at http://127.3.131.130:42145/ using document root <none> and password file <none>
I20260621 05:14:24.570433 22950 fs_manager.cc:359] Metadata directory not provided
I20260621 05:14:24.570480 22950 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 05:14:24.570640 22950 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 05:14:24.571767 22950 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782018616648578-3598-0/minicluster-data/ts-1/data/instance:
uuid: "55344868c5da4ad3be557a66159f268c"
format_stamp: "Formatted at 2026-06-21 05:14:24 on dist-test-slave-vrq1"
server_key: "3d1372ff7b2cf250b9691f5be3475200"
server_key_iv: "63bc5b8a21ba73460dc489bfa59bd0f3"
server_key_version: "encryptionkey@0"
I20260621 05:14:24.572116 22950 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782018616648578-3598-0/minicluster-data/ts-1/wal/instance:
uuid: "55344868c5da4ad3be557a66159f268c"
format_stamp: "Formatted at 2026-06-21 05:14:24 on dist-test-slave-vrq1"
server_key: "3d1372ff7b2cf250b9691f5be3475200"
server_key_iv: "63bc5b8a21ba73460dc489bfa59bd0f3"
server_key_version: "encryptionkey@0"
I20260621 05:14:24.573447 22950 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260621 05:14:24.574189 22965 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 05:14:24.574361 22950 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260621 05:14:24.574458 22950 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782018616648578-3598-0/minicluster-data/ts-1/data,/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782018616648578-3598-0/minicluster-data/ts-1/wal
uuid: "55344868c5da4ad3be557a66159f268c"
format_stamp: "Formatted at 2026-06-21 05:14:24 on dist-test-slave-vrq1"
server_key: "3d1372ff7b2cf250b9691f5be3475200"
server_key_iv: "63bc5b8a21ba73460dc489bfa59bd0f3"
server_key_version: "encryptionkey@0"
I20260621 05:14:24.574527 22950 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782018616648578-3598-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782018616648578-3598-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782018616648578-3598-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 05:14:24.587548 22950 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 05:14:24.589480 22950 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 05:14:24.589644 22950 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 05:14:24.589881 22950 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 05:14:24.590194 22950 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 05:14:24.590247 22950 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 05:14:24.590289 22950 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 05:14:24.590317 22950 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 05:14:24.596499 22950 rpc_server.cc:307] RPC server started. Bound to: 127.3.131.130:41983
I20260621 05:14:24.596567 23078 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.131.130:41983 every 8 connection(s)
I20260621 05:14:24.596910 22950 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782018616648578-3598-0/minicluster-data/ts-1/data/info.pb
Jun 21 05:14:24 dist-test-slave-vrq1 krb5kdc[22709](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782018864, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.131.130@KRBTEST.COM for kudu/127.3.131.190@KRBTEST.COM
I20260621 05:14:24.606228 3598 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWZOx2_/build/release/bin/kudu as pid 22950
I20260621 05:14:24.606374 3598 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782018616648578-3598-0/minicluster-data/ts-1/wal/instance
I20260621 05:14:24.606637 3598 external_mini_cluster.cc:1428] Setting key 173958d55106d87a93433571c96d782a
I20260621 05:14:24.607076 22794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:14:24.597635 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.131.130:45883 (local address 127.3.131.190:44125)
0621 05:14:24.597757 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0621 05:14:24.597760 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0621 05:14:24.597902 (+ 142us) server_negotiation.cc:407] Connection header received
0621 05:14:24.598385 (+ 483us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:14:24.598386 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:14:24.598397 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:14:24.598452 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:14:24.599530 (+ 1078us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:14:24.600091 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:14:24.601143 (+ 1052us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:14:24.601344 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:14:24.601411 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:14:24.603928 (+ 2517us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:14:24.603929 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:14:24.603929 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:14:24.603936 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:14:24.605417 (+ 1481us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:14:24.605865 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:14:24.605867 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:14:24.605867 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:14:24.605943 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:14:24.606238 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:14:24.606240 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:14:24.606240 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:14:24.606408 (+ 168us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:14:24.606490 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0621 05:14:24.606859 (+ 369us) server_negotiation.cc:299] Negotiation successful
0621 05:14:24.606967 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260621 05:14:24.607366 23081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:14:24.597610 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.131.190:44125 (local address 127.3.131.130:45883)
0621 05:14:24.597783 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0621 05:14:24.597821 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0621 05:14:24.598289 (+ 468us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:14:24.598502 (+ 213us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:14:24.598505 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:14:24.599119 (+ 614us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:14:24.599419 (+ 300us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:14:24.599425 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:14:24.600213 (+ 788us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:14:24.600214 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:14:24.601038 (+ 824us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:14:24.601041 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:14:24.601448 (+ 407us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:14:24.601449 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:14:24.601579 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:14:24.602124 (+ 545us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 05:14:24.602132 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:14:24.603838 (+ 1706us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 05:14:24.605502 (+ 1664us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:14:24.605508 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:14:24.605511 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:14:24.605773 (+ 262us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:14:24.606022 (+ 249us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:14:24.606023 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:14:24.606023 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:14:24.606146 (+ 123us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:14:24.606494 (+ 348us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 05:14:24.606502 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 05:14:24.606700 (+ 198us) client_negotiation.cc:769] Sending connection context
0621 05:14:24.606819 (+ 119us) client_negotiation.cc:240] Negotiation successful
0621 05:14:24.607044 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":100,"threads_started":1}
I20260621 05:14:24.607818 23079 heartbeater.cc:347] Connected to a master server at 127.3.131.190:44125
I20260621 05:14:24.607929 23079 heartbeater.cc:464] Registering TS with master...
I20260621 05:14:24.608127 23079 heartbeater.cc:511] Master 127.3.131.190:44125 requested a full tablet report, sending...
I20260621 05:14:24.608493 22754 ts_manager.cc:194] Registered new tserver with Master: 55344868c5da4ad3be557a66159f268c (127.3.131.130:41983)
I20260621 05:14:24.608892 22754 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.131.130@KRBTEST.COM'} at 127.3.131.130:45883
WARNING: no policy specified for kudu/127.3.131.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.131.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.131.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782018616648578-3598-0/krb5kdc/kudu_127.3.131.131.keytab.
Entry for principal kudu/127.3.131.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782018616648578-3598-0/krb5kdc/kudu_127.3.131.131.keytab.
I20260621 05:14:24.634408 3598 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWZOx2_/build/release/bin/kudu
/tmp/dist-test-taskWZOx2_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782018616648578-3598-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782018616648578-3598-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782018616648578-3598-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782018616648578-3598-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.131.131:0
--local_ip_for_outbound_sockets=127.3.131.131
--webserver_interface=127.3.131.131
--webserver_port=0
--tserver_master_addrs=127.3.131.190:44125
--keytab_file=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782018616648578-3598-0/krb5kdc/kudu_127.3.131.131.keytab
--principal=kudu/127.3.131.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.131.148:40953
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782018616648578-3598-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782018616648578-3598-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782018616648578-3598-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 05:14:24.716051 23084 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 05:14:24.716336 23084 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 05:14:24.716363 23084 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 05:14:24.717514 23084 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 05:14:24.717554 23084 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 05:14:24.717581 23084 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.131.131
I20260621 05:14:24.719030 23084 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.131.148:40953
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782018616648578-3598-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782018616648578-3598-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782018616648578-3598-0/krb5kdc/kudu_127.3.131.131.keytab
--principal=kudu/127.3.131.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.131.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782018616648578-3598-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.131.131
--webserver_port=0
--tserver_master_addrs=127.3.131.190:44125
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23084
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.131.131
--log_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782018616648578-3598-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 21 Jun 2026 04:16:16 UTC on ccycloud.quasar-xycaha.root.comops.site
build id 4457881
I20260621 05:14:24.719275 23084 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 05:14:24.719470 23084 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 05:14:24.724946 23089 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:14:24.724979 23092 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:14:24.724968 23090 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:14:24.725204 23084 server_base.cc:1034] running on GCE node
I20260621 05:14:24.725553 23084 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 05:14:24.725831 23084 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 05:14:24.727008 23084 hybrid_clock.cc:648] HybridClock initialized: now 1782018864726993 us; error 29 us; skew 500 ppm
Jun 21 05:14:24 dist-test-slave-vrq1 krb5kdc[22709](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782018864, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.131.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 05:14:24.729825 23084 init.cc:377] Logged in from keytab as kudu/127.3.131.131@KRBTEST.COM (short username kudu)
I20260621 05:14:24.730271 23084 webserver.cc:460] Webserver started at http://127.3.131.131:41997/ using document root <none> and password file <none>
I20260621 05:14:24.730530 23084 fs_manager.cc:359] Metadata directory not provided
I20260621 05:14:24.730580 23084 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 05:14:24.730715 23084 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 05:14:24.731947 23084 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782018616648578-3598-0/minicluster-data/ts-2/data/instance:
uuid: "9e366e53c2474cd3ad5944198c93d7f8"
format_stamp: "Formatted at 2026-06-21 05:14:24 on dist-test-slave-vrq1"
server_key: "5ab01cf0c3bbcf4d7666b3c456573f3f"
server_key_iv: "37be237a54ff9879eb0898227c3088d0"
server_key_version: "encryptionkey@0"
I20260621 05:14:24.732276 23084 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782018616648578-3598-0/minicluster-data/ts-2/wal/instance:
uuid: "9e366e53c2474cd3ad5944198c93d7f8"
format_stamp: "Formatted at 2026-06-21 05:14:24 on dist-test-slave-vrq1"
server_key: "5ab01cf0c3bbcf4d7666b3c456573f3f"
server_key_iv: "37be237a54ff9879eb0898227c3088d0"
server_key_version: "encryptionkey@0"
I20260621 05:14:24.733835 23084 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260621 05:14:24.734767 23099 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 05:14:24.734970 23084 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260621 05:14:24.735072 23084 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782018616648578-3598-0/minicluster-data/ts-2/data,/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782018616648578-3598-0/minicluster-data/ts-2/wal
uuid: "9e366e53c2474cd3ad5944198c93d7f8"
format_stamp: "Formatted at 2026-06-21 05:14:24 on dist-test-slave-vrq1"
server_key: "5ab01cf0c3bbcf4d7666b3c456573f3f"
server_key_iv: "37be237a54ff9879eb0898227c3088d0"
server_key_version: "encryptionkey@0"
I20260621 05:14:24.735157 23084 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782018616648578-3598-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782018616648578-3598-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782018616648578-3598-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 05:14:24.759792 23084 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 05:14:24.762602 23084 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 05:14:24.762804 23084 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 05:14:24.763082 23084 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 05:14:24.763463 23084 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 05:14:24.763530 23084 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 05:14:24.763579 23084 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 05:14:24.763625 23084 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 05:14:24.770588 23084 rpc_server.cc:307] RPC server started. Bound to: 127.3.131.131:46363
I20260621 05:14:24.770658 23212 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.131.131:46363 every 8 connection(s)
I20260621 05:14:24.771068 23084 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782018616648578-3598-0/minicluster-data/ts-2/data/info.pb
Jun 21 05:14:24 dist-test-slave-vrq1 krb5kdc[22709](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782018864, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.131.131@KRBTEST.COM for kudu/127.3.131.190@KRBTEST.COM
I20260621 05:14:24.780066 3598 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWZOx2_/build/release/bin/kudu as pid 23084
I20260621 05:14:24.780448 3598 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782018616648578-3598-0/minicluster-data/ts-2/wal/instance
I20260621 05:14:24.780720 3598 external_mini_cluster.cc:1428] Setting key 709a36dae991e5675c4c99ee7c7d1515
I20260621 05:14:24.782256 22794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:14:24.772012 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.131.131:41983 (local address 127.3.131.190:44125)
0621 05:14:24.772105 (+ 93us) server_negotiation.cc:206] Beginning negotiation
0621 05:14:24.772107 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:14:24.772302 (+ 195us) server_negotiation.cc:407] Connection header received
0621 05:14:24.773282 (+ 980us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:14:24.773283 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:14:24.773294 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:14:24.773360 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:14:24.774036 (+ 676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:14:24.774548 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:14:24.775740 (+ 1192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:14:24.775952 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:14:24.776013 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:14:24.778745 (+ 2732us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:14:24.778746 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:14:24.778747 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:14:24.778756 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:14:24.780611 (+ 1855us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:14:24.781192 (+ 581us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:14:24.781193 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:14:24.781194 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:14:24.781273 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:14:24.781598 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:14:24.781599 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:14:24.781599 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:14:24.781766 (+ 167us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:14:24.781829 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0621 05:14:24.782093 (+ 264us) server_negotiation.cc:299] Negotiation successful
0621 05:14:24.782189 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260621 05:14:24.782464 23215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:14:24.771960 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.131.190:44125 (local address 127.3.131.131:41983)
0621 05:14:24.772136 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0621 05:14:24.772182 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0621 05:14:24.773164 (+ 982us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:14:24.773420 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:14:24.773423 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:14:24.773721 (+ 298us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:14:24.773931 (+ 210us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:14:24.773935 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:14:24.774666 (+ 731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:14:24.774667 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:14:24.775631 (+ 964us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:14:24.775635 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:14:24.776050 (+ 415us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:14:24.776051 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:14:24.776179 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:14:24.776820 (+ 641us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 05:14:24.776829 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:14:24.778612 (+ 1783us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 05:14:24.780765 (+ 2153us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:14:24.780776 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:14:24.780779 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:14:24.781125 (+ 346us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:14:24.781362 (+ 237us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:14:24.781363 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:14:24.781364 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:14:24.781520 (+ 156us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:14:24.781832 (+ 312us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 05:14:24.781837 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 05:14:24.781971 (+ 134us) client_negotiation.cc:769] Sending connection context
0621 05:14:24.782063 (+ 92us) client_negotiation.cc:240] Negotiation successful
0621 05:14:24.782211 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":96,"threads_started":1}
I20260621 05:14:24.782960 23213 heartbeater.cc:347] Connected to a master server at 127.3.131.190:44125
I20260621 05:14:24.783064 23213 heartbeater.cc:464] Registering TS with master...
I20260621 05:14:24.783282 23213 heartbeater.cc:511] Master 127.3.131.190:44125 requested a full tablet report, sending...
I20260621 05:14:24.783651 22754 ts_manager.cc:194] Registered new tserver with Master: 9e366e53c2474cd3ad5944198c93d7f8 (127.3.131.131:46363)
I20260621 05:14:24.784019 22754 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.131.131@KRBTEST.COM'} at 127.3.131.131:41983
I20260621 05:14:24.792869 3598 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260621 05:14:24.801146 22794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:14:24.794435 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37916 (local address 127.3.131.190:44125)
0621 05:14:24.794537 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0621 05:14:24.794540 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0621 05:14:24.794638 (+ 98us) server_negotiation.cc:407] Connection header received
0621 05:14:24.794789 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:14:24.794790 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:14:24.794801 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:14:24.794880 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:14:24.795584 (+ 704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:14:24.796100 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:14:24.796874 (+ 774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:14:24.797089 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:14:24.797171 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:14:24.798456 (+ 1285us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:14:24.798457 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:14:24.798458 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:14:24.798465 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:14:24.799915 (+ 1450us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:14:24.800293 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:14:24.800294 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:14:24.800294 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:14:24.800353 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:14:24.800550 (+ 197us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:14:24.800550 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:14:24.800551 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:14:24.800719 (+ 168us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:14:24.800821 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0621 05:14:24.800947 (+ 126us) server_negotiation.cc:299] Negotiation successful
0621 05:14:24.801034 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260621 05:14:24.803447 22754 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37916:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260621 05:14:24.805186 22754 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260621 05:14:24.813264 23229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:14:24.808953 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49078 (local address 127.3.131.129:39779)
0621 05:14:24.809163 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0621 05:14:24.809168 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 05:14:24.809179 (+ 11us) server_negotiation.cc:407] Connection header received
0621 05:14:24.809224 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:14:24.809226 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:14:24.809242 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:14:24.809547 (+ 305us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 05:14:24.809626 (+ 79us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:14:24.810745 (+ 1119us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:14:24.812266 (+ 1521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:14:24.812810 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:14:24.812896 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:14:24.812962 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0621 05:14:24.813120 (+ 158us) server_negotiation.cc:299] Negotiation successful
0621 05:14:24.813165 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":179,"thread_start_us":94,"threads_started":1}
I20260621 05:14:24.813341 23228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:14:24.808807 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.131.129:39779 (local address 127.0.0.1:49078)
0621 05:14:24.809011 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0621 05:14:24.809022 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0621 05:14:24.809074 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:14:24.809351 (+ 277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:14:24.809351 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:14:24.809354 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 05:14:24.809424 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:14:24.809466 (+ 42us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:14:24.810832 (+ 1366us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:14:24.810833 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:14:24.812149 (+ 1316us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:14:24.812152 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:14:24.812896 (+ 744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:14:24.812897 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:14:24.812981 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:14:24.812996 (+ 15us) client_negotiation.cc:769] Sending connection context
0621 05:14:24.813101 (+ 105us) client_negotiation.cc:240] Negotiation successful
0621 05:14:24.813165 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":180,"mutex_wait_us":68,"thread_start_us":80,"threads_started":1}
I20260621 05:14:24.813485 23226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:14:24.808586 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.131.131:46363 (local address 127.0.0.1:40982)
0621 05:14:24.808745 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0621 05:14:24.808757 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0621 05:14:24.808803 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:14:24.809186 (+ 383us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:14:24.809190 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:14:24.809193 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 05:14:24.809327 (+ 134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:14:24.809329 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:14:24.810495 (+ 1166us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:14:24.810496 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:14:24.812181 (+ 1685us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:14:24.812183 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:14:24.813236 (+ 1053us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:14:24.813237 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:14:24.813303 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:14:24.813307 (+ 4us) client_negotiation.cc:769] Sending connection context
0621 05:14:24.813341 (+ 34us) client_negotiation.cc:240] Negotiation successful
0621 05:14:24.813391 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":136,"mutex_wait_us":3,"thread_start_us":88,"threads_started":1}
I20260621 05:14:24.813858 23227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:14:24.808640 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40982 (local address 127.3.131.131:46363)
0621 05:14:24.808937 (+ 297us) server_negotiation.cc:206] Beginning negotiation
0621 05:14:24.808943 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 05:14:24.808960 (+ 17us) server_negotiation.cc:407] Connection header received
0621 05:14:24.809037 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:14:24.809040 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:14:24.809059 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:14:24.809421 (+ 362us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 05:14:24.809512 (+ 91us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:14:24.810379 (+ 867us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:14:24.812300 (+ 1921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:14:24.813120 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:14:24.813597 (+ 477us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:14:24.813672 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0621 05:14:24.813701 (+ 29us) server_negotiation.cc:299] Negotiation successful
0621 05:14:24.813762 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":256,"thread_start_us":85,"threads_started":1}
I20260621 05:14:24.814560 23224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:14:24.808281 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.131.130:41983 (local address 127.0.0.1:48752)
0621 05:14:24.808446 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0621 05:14:24.808498 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0621 05:14:24.808624 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:14:24.809355 (+ 731us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:14:24.809357 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:14:24.809359 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 05:14:24.809463 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:14:24.809465 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:14:24.811907 (+ 2442us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:14:24.811908 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:14:24.813158 (+ 1250us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:14:24.813161 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:14:24.814301 (+ 1140us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:14:24.814302 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:14:24.814355 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:14:24.814358 (+ 3us) client_negotiation.cc:769] Sending connection context
0621 05:14:24.814437 (+ 79us) client_negotiation.cc:240] Negotiation successful
0621 05:14:24.814483 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":103,"threads_started":1}
I20260621 05:14:24.814584 23225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:14:24.808416 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48752 (local address 127.3.131.130:41983)
0621 05:14:24.809208 (+ 792us) server_negotiation.cc:206] Beginning negotiation
0621 05:14:24.809216 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0621 05:14:24.809224 (+ 8us) server_negotiation.cc:407] Connection header received
0621 05:14:24.809261 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:14:24.809264 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:14:24.809280 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:14:24.809357 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 05:14:24.809540 (+ 183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:14:24.810731 (+ 1191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:14:24.813280 (+ 2549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:14:24.814123 (+ 843us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:14:24.814202 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:14:24.814268 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0621 05:14:24.814454 (+ 186us) server_negotiation.cc:299] Negotiation successful
0621 05:14:24.814497 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":748,"thread_start_us":115,"threads_started":1}
I20260621 05:14:24.814620 23147 tablet_service.cc:1467] Processing CreateTablet for tablet 7e1771f9c37e4a9bb8de117287615839 (DEFAULT_TABLE table=test-table [id=6ee86190165c4ea5825860e59b19ef3c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 05:14:24.814620 22879 tablet_service.cc:1467] Processing CreateTablet for tablet 7e1771f9c37e4a9bb8de117287615839 (DEFAULT_TABLE table=test-table [id=6ee86190165c4ea5825860e59b19ef3c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 05:14:24.814985 23147 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7e1771f9c37e4a9bb8de117287615839. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 05:14:24.814985 22879 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7e1771f9c37e4a9bb8de117287615839. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 05:14:24.815984 23013 tablet_service.cc:1467] Processing CreateTablet for tablet 7e1771f9c37e4a9bb8de117287615839 (DEFAULT_TABLE table=test-table [id=6ee86190165c4ea5825860e59b19ef3c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 05:14:24.816322 23013 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7e1771f9c37e4a9bb8de117287615839. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 05:14:24.818497 23230 tablet_bootstrap.cc:492] T 7e1771f9c37e4a9bb8de117287615839 P 9e366e53c2474cd3ad5944198c93d7f8: Bootstrap starting.
I20260621 05:14:24.819061 23232 tablet_bootstrap.cc:492] T 7e1771f9c37e4a9bb8de117287615839 P 55344868c5da4ad3be557a66159f268c: Bootstrap starting.
I20260621 05:14:24.819310 23230 tablet_bootstrap.cc:654] T 7e1771f9c37e4a9bb8de117287615839 P 9e366e53c2474cd3ad5944198c93d7f8: Neither blocks nor log segments found. Creating new log.
I20260621 05:14:24.819798 23232 tablet_bootstrap.cc:654] T 7e1771f9c37e4a9bb8de117287615839 P 55344868c5da4ad3be557a66159f268c: Neither blocks nor log segments found. Creating new log.
I20260621 05:14:24.819988 23230 log.cc:824] T 7e1771f9c37e4a9bb8de117287615839 P 9e366e53c2474cd3ad5944198c93d7f8: Log is configured to *not* fsync() on all Append() calls
I20260621 05:14:24.820365 23232 log.cc:824] T 7e1771f9c37e4a9bb8de117287615839 P 55344868c5da4ad3be557a66159f268c: Log is configured to *not* fsync() on all Append() calls
I20260621 05:14:24.820396 23231 tablet_bootstrap.cc:492] T 7e1771f9c37e4a9bb8de117287615839 P fe661fee975b4b2587e1f23dcdbddf9b: Bootstrap starting.
I20260621 05:14:24.821017 23231 tablet_bootstrap.cc:654] T 7e1771f9c37e4a9bb8de117287615839 P fe661fee975b4b2587e1f23dcdbddf9b: Neither blocks nor log segments found. Creating new log.
I20260621 05:14:24.821118 23232 tablet_bootstrap.cc:492] T 7e1771f9c37e4a9bb8de117287615839 P 55344868c5da4ad3be557a66159f268c: No bootstrap required, opened a new log
I20260621 05:14:24.821190 23232 ts_tablet_manager.cc:1397] T 7e1771f9c37e4a9bb8de117287615839 P 55344868c5da4ad3be557a66159f268c: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260621 05:14:24.821588 23231 log.cc:824] T 7e1771f9c37e4a9bb8de117287615839 P fe661fee975b4b2587e1f23dcdbddf9b: Log is configured to *not* fsync() on all Append() calls
I20260621 05:14:24.822152 23230 tablet_bootstrap.cc:492] T 7e1771f9c37e4a9bb8de117287615839 P 9e366e53c2474cd3ad5944198c93d7f8: No bootstrap required, opened a new log
I20260621 05:14:24.822227 23230 ts_tablet_manager.cc:1397] T 7e1771f9c37e4a9bb8de117287615839 P 9e366e53c2474cd3ad5944198c93d7f8: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260621 05:14:24.822494 23231 tablet_bootstrap.cc:492] T 7e1771f9c37e4a9bb8de117287615839 P fe661fee975b4b2587e1f23dcdbddf9b: No bootstrap required, opened a new log
I20260621 05:14:24.822573 23231 ts_tablet_manager.cc:1397] T 7e1771f9c37e4a9bb8de117287615839 P fe661fee975b4b2587e1f23dcdbddf9b: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260621 05:14:24.823330 23232 raft_consensus.cc:348] T 7e1771f9c37e4a9bb8de117287615839 P 55344868c5da4ad3be557a66159f268c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55344868c5da4ad3be557a66159f268c" member_type: VOTER last_known_addr { host: "127.3.131.130" port: 41983 } } peers { permanent_uuid: "9e366e53c2474cd3ad5944198c93d7f8" member_type: VOTER last_known_addr { host: "127.3.131.131" port: 46363 } } peers { permanent_uuid: "fe661fee975b4b2587e1f23dcdbddf9b" member_type: VOTER last_known_addr { host: "127.3.131.129" port: 39779 } }
I20260621 05:14:24.823518 23232 raft_consensus.cc:374] T 7e1771f9c37e4a9bb8de117287615839 P 55344868c5da4ad3be557a66159f268c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 05:14:24.823555 23232 raft_consensus.cc:729] T 7e1771f9c37e4a9bb8de117287615839 P 55344868c5da4ad3be557a66159f268c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 55344868c5da4ad3be557a66159f268c, State: Initialized, Role: FOLLOWER
I20260621 05:14:24.823683 23232 consensus_queue.cc:260] T 7e1771f9c37e4a9bb8de117287615839 P 55344868c5da4ad3be557a66159f268c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55344868c5da4ad3be557a66159f268c" member_type: VOTER last_known_addr { host: "127.3.131.130" port: 41983 } } peers { permanent_uuid: "9e366e53c2474cd3ad5944198c93d7f8" member_type: VOTER last_known_addr { host: "127.3.131.131" port: 46363 } } peers { permanent_uuid: "fe661fee975b4b2587e1f23dcdbddf9b" member_type: VOTER last_known_addr { host: "127.3.131.129" port: 39779 } }
I20260621 05:14:24.823957 23232 ts_tablet_manager.cc:1428] T 7e1771f9c37e4a9bb8de117287615839 P 55344868c5da4ad3be557a66159f268c: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260621 05:14:24.824092 23079 heartbeater.cc:503] Master 127.3.131.190:44125 was elected leader, sending a full tablet report...
I20260621 05:14:24.824281 23230 raft_consensus.cc:348] T 7e1771f9c37e4a9bb8de117287615839 P 9e366e53c2474cd3ad5944198c93d7f8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55344868c5da4ad3be557a66159f268c" member_type: VOTER last_known_addr { host: "127.3.131.130" port: 41983 } } peers { permanent_uuid: "9e366e53c2474cd3ad5944198c93d7f8" member_type: VOTER last_known_addr { host: "127.3.131.131" port: 46363 } } peers { permanent_uuid: "fe661fee975b4b2587e1f23dcdbddf9b" member_type: VOTER last_known_addr { host: "127.3.131.129" port: 39779 } }
I20260621 05:14:24.824316 23231 raft_consensus.cc:348] T 7e1771f9c37e4a9bb8de117287615839 P fe661fee975b4b2587e1f23dcdbddf9b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55344868c5da4ad3be557a66159f268c" member_type: VOTER last_known_addr { host: "127.3.131.130" port: 41983 } } peers { permanent_uuid: "9e366e53c2474cd3ad5944198c93d7f8" member_type: VOTER last_known_addr { host: "127.3.131.131" port: 46363 } } peers { permanent_uuid: "fe661fee975b4b2587e1f23dcdbddf9b" member_type: VOTER last_known_addr { host: "127.3.131.129" port: 39779 } }
I20260621 05:14:24.824458 23230 raft_consensus.cc:374] T 7e1771f9c37e4a9bb8de117287615839 P 9e366e53c2474cd3ad5944198c93d7f8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 05:14:24.824476 23231 raft_consensus.cc:374] T 7e1771f9c37e4a9bb8de117287615839 P fe661fee975b4b2587e1f23dcdbddf9b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 05:14:24.824506 23230 raft_consensus.cc:729] T 7e1771f9c37e4a9bb8de117287615839 P 9e366e53c2474cd3ad5944198c93d7f8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9e366e53c2474cd3ad5944198c93d7f8, State: Initialized, Role: FOLLOWER
I20260621 05:14:24.824512 23231 raft_consensus.cc:729] T 7e1771f9c37e4a9bb8de117287615839 P fe661fee975b4b2587e1f23dcdbddf9b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe661fee975b4b2587e1f23dcdbddf9b, State: Initialized, Role: FOLLOWER
I20260621 05:14:24.824620 23231 consensus_queue.cc:260] T 7e1771f9c37e4a9bb8de117287615839 P fe661fee975b4b2587e1f23dcdbddf9b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55344868c5da4ad3be557a66159f268c" member_type: VOTER last_known_addr { host: "127.3.131.130" port: 41983 } } peers { permanent_uuid: "9e366e53c2474cd3ad5944198c93d7f8" member_type: VOTER last_known_addr { host: "127.3.131.131" port: 46363 } } peers { permanent_uuid: "fe661fee975b4b2587e1f23dcdbddf9b" member_type: VOTER last_known_addr { host: "127.3.131.129" port: 39779 } }
I20260621 05:14:24.824704 23230 consensus_queue.cc:260] T 7e1771f9c37e4a9bb8de117287615839 P 9e366e53c2474cd3ad5944198c93d7f8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55344868c5da4ad3be557a66159f268c" member_type: VOTER last_known_addr { host: "127.3.131.130" port: 41983 } } peers { permanent_uuid: "9e366e53c2474cd3ad5944198c93d7f8" member_type: VOTER last_known_addr { host: "127.3.131.131" port: 46363 } } peers { permanent_uuid: "fe661fee975b4b2587e1f23dcdbddf9b" member_type: VOTER last_known_addr { host: "127.3.131.129" port: 39779 } }
I20260621 05:14:24.824903 23231 ts_tablet_manager.cc:1428] T 7e1771f9c37e4a9bb8de117287615839 P fe661fee975b4b2587e1f23dcdbddf9b: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260621 05:14:24.825003 23230 ts_tablet_manager.cc:1428] T 7e1771f9c37e4a9bb8de117287615839 P 9e366e53c2474cd3ad5944198c93d7f8: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260621 05:14:24.825024 22945 heartbeater.cc:503] Master 127.3.131.190:44125 was elected leader, sending a full tablet report...
I20260621 05:14:24.825137 23213 heartbeater.cc:503] Master 127.3.131.190:44125 was elected leader, sending a full tablet report...
W20260621 05:14:24.847496 23080 tablet.cc:2369] T 7e1771f9c37e4a9bb8de117287615839 P 55344868c5da4ad3be557a66159f268c: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260621 05:14:24.934352 22946 tablet.cc:2369] T 7e1771f9c37e4a9bb8de117287615839 P fe661fee975b4b2587e1f23dcdbddf9b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260621 05:14:24.964396 23238 raft_consensus.cc:482] T 7e1771f9c37e4a9bb8de117287615839 P 9e366e53c2474cd3ad5944198c93d7f8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260621 05:14:24.964520 23238 raft_consensus.cc:504] T 7e1771f9c37e4a9bb8de117287615839 P 9e366e53c2474cd3ad5944198c93d7f8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55344868c5da4ad3be557a66159f268c" member_type: VOTER last_known_addr { host: "127.3.131.130" port: 41983 } } peers { permanent_uuid: "9e366e53c2474cd3ad5944198c93d7f8" member_type: VOTER last_known_addr { host: "127.3.131.131" port: 46363 } } peers { permanent_uuid: "fe661fee975b4b2587e1f23dcdbddf9b" member_type: VOTER last_known_addr { host: "127.3.131.129" port: 39779 } }
I20260621 05:14:24.964854 23238 leader_election.cc:288] T 7e1771f9c37e4a9bb8de117287615839 P 9e366e53c2474cd3ad5944198c93d7f8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 55344868c5da4ad3be557a66159f268c (127.3.131.130:41983), fe661fee975b4b2587e1f23dcdbddf9b (127.3.131.129:39779)
I20260621 05:14:24.968428 23229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:14:24.965315 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.131.131:37765 (local address 127.3.131.129:39779)
0621 05:14:24.965363 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0621 05:14:24.965366 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0621 05:14:24.965467 (+ 101us) server_negotiation.cc:407] Connection header received
0621 05:14:24.965533 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:14:24.965534 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:14:24.965543 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:14:24.965657 (+ 114us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 05:14:24.965808 (+ 151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:14:24.966260 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:14:24.967695 (+ 1435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:14:24.968056 (+ 361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:14:24.968145 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:14:24.968153 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0621 05:14:24.968320 (+ 167us) server_negotiation.cc:299] Negotiation successful
0621 05:14:24.968355 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260621 05:14:24.968438 23239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:14:24.965189 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.131.129:39779 (local address 127.3.131.131:37765)
0621 05:14:24.965399 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0621 05:14:24.965408 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0621 05:14:24.965479 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:14:24.965663 (+ 184us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:14:24.965664 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:14:24.965665 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 05:14:24.965741 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:14:24.965742 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:14:24.966338 (+ 596us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:14:24.966339 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:14:24.967603 (+ 1264us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:14:24.967606 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:14:24.968146 (+ 540us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:14:24.968147 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:14:24.968250 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:14:24.968254 (+ 4us) client_negotiation.cc:769] Sending connection context
0621 05:14:24.968313 (+ 59us) client_negotiation.cc:240] Negotiation successful
0621 05:14:24.968352 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":196,"thread_start_us":133,"threads_started":1}
I20260621 05:14:24.968590 23225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:14:24.965137 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.131.131:36405 (local address 127.3.131.130:41983)
0621 05:14:24.965201 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0621 05:14:24.965203 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:14:24.965216 (+ 13us) server_negotiation.cc:407] Connection header received
0621 05:14:24.965253 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:14:24.965254 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:14:24.965265 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:14:24.965374 (+ 109us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 05:14:24.965549 (+ 175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:14:24.966092 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:14:24.967766 (+ 1674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:14:24.968198 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:14:24.968294 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:14:24.968303 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0621 05:14:24.968465 (+ 162us) server_negotiation.cc:299] Negotiation successful
0621 05:14:24.968504 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260621 05:14:24.968613 23215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:14:24.965029 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.131.130:41983 (local address 127.3.131.131:36405)
0621 05:14:24.965086 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0621 05:14:24.965099 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0621 05:14:24.965150 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:14:24.965381 (+ 231us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:14:24.965382 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:14:24.965385 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 05:14:24.965470 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:14:24.965472 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:14:24.966185 (+ 713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:14:24.966186 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:14:24.967642 (+ 1456us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:14:24.967645 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:14:24.968290 (+ 645us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:14:24.968291 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:14:24.968384 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:14:24.968388 (+ 4us) client_negotiation.cc:769] Sending connection context
0621 05:14:24.968459 (+ 71us) client_negotiation.cc:240] Negotiation successful
0621 05:14:24.968504 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":34,"mutex_wait_us":40}
I20260621 05:14:24.968777 22899 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e1771f9c37e4a9bb8de117287615839" candidate_uuid: "9e366e53c2474cd3ad5944198c93d7f8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe661fee975b4b2587e1f23dcdbddf9b" is_pre_election: true
I20260621 05:14:24.968977 22899 raft_consensus.cc:2454] T 7e1771f9c37e4a9bb8de117287615839 P fe661fee975b4b2587e1f23dcdbddf9b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9e366e53c2474cd3ad5944198c93d7f8 in term 0.
I20260621 05:14:24.968926 23033 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e1771f9c37e4a9bb8de117287615839" candidate_uuid: "9e366e53c2474cd3ad5944198c93d7f8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "55344868c5da4ad3be557a66159f268c" is_pre_election: true
I20260621 05:14:24.969095 23033 raft_consensus.cc:2454] T 7e1771f9c37e4a9bb8de117287615839 P 55344868c5da4ad3be557a66159f268c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9e366e53c2474cd3ad5944198c93d7f8 in term 0.
I20260621 05:14:24.969213 23103 leader_election.cc:302] T 7e1771f9c37e4a9bb8de117287615839 P 9e366e53c2474cd3ad5944198c93d7f8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9e366e53c2474cd3ad5944198c93d7f8, fe661fee975b4b2587e1f23dcdbddf9b; no voters:
I20260621 05:14:24.969354 23238 raft_consensus.cc:2781] T 7e1771f9c37e4a9bb8de117287615839 P 9e366e53c2474cd3ad5944198c93d7f8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260621 05:14:24.969416 23238 raft_consensus.cc:482] T 7e1771f9c37e4a9bb8de117287615839 P 9e366e53c2474cd3ad5944198c93d7f8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260621 05:14:24.969435 23238 raft_consensus.cc:3037] T 7e1771f9c37e4a9bb8de117287615839 P 9e366e53c2474cd3ad5944198c93d7f8 [term 0 FOLLOWER]: Advancing to term 1
I20260621 05:14:24.970221 23238 raft_consensus.cc:504] T 7e1771f9c37e4a9bb8de117287615839 P 9e366e53c2474cd3ad5944198c93d7f8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55344868c5da4ad3be557a66159f268c" member_type: VOTER last_known_addr { host: "127.3.131.130" port: 41983 } } peers { permanent_uuid: "9e366e53c2474cd3ad5944198c93d7f8" member_type: VOTER last_known_addr { host: "127.3.131.131" port: 46363 } } peers { permanent_uuid: "fe661fee975b4b2587e1f23dcdbddf9b" member_type: VOTER last_known_addr { host: "127.3.131.129" port: 39779 } }
I20260621 05:14:24.970345 23238 leader_election.cc:288] T 7e1771f9c37e4a9bb8de117287615839 P 9e366e53c2474cd3ad5944198c93d7f8 [CANDIDATE]: Term 1 election: Requested vote from peers 55344868c5da4ad3be557a66159f268c (127.3.131.130:41983), fe661fee975b4b2587e1f23dcdbddf9b (127.3.131.129:39779)
I20260621 05:14:24.970700 23033 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e1771f9c37e4a9bb8de117287615839" candidate_uuid: "9e366e53c2474cd3ad5944198c93d7f8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "55344868c5da4ad3be557a66159f268c"
I20260621 05:14:24.970710 22899 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e1771f9c37e4a9bb8de117287615839" candidate_uuid: "9e366e53c2474cd3ad5944198c93d7f8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe661fee975b4b2587e1f23dcdbddf9b"
I20260621 05:14:24.970786 23033 raft_consensus.cc:3037] T 7e1771f9c37e4a9bb8de117287615839 P 55344868c5da4ad3be557a66159f268c [term 0 FOLLOWER]: Advancing to term 1
I20260621 05:14:24.970805 22899 raft_consensus.cc:3037] T 7e1771f9c37e4a9bb8de117287615839 P fe661fee975b4b2587e1f23dcdbddf9b [term 0 FOLLOWER]: Advancing to term 1
I20260621 05:14:24.971611 22899 raft_consensus.cc:2454] T 7e1771f9c37e4a9bb8de117287615839 P fe661fee975b4b2587e1f23dcdbddf9b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9e366e53c2474cd3ad5944198c93d7f8 in term 1.
I20260621 05:14:24.971666 23033 raft_consensus.cc:2454] T 7e1771f9c37e4a9bb8de117287615839 P 55344868c5da4ad3be557a66159f268c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9e366e53c2474cd3ad5944198c93d7f8 in term 1.
I20260621 05:14:24.971858 23103 leader_election.cc:302] T 7e1771f9c37e4a9bb8de117287615839 P 9e366e53c2474cd3ad5944198c93d7f8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 55344868c5da4ad3be557a66159f268c, 9e366e53c2474cd3ad5944198c93d7f8; no voters:
I20260621 05:14:24.972031 23238 raft_consensus.cc:2781] T 7e1771f9c37e4a9bb8de117287615839 P 9e366e53c2474cd3ad5944198c93d7f8 [term 1 FOLLOWER]: Leader election won for term 1
I20260621 05:14:24.972227 23238 raft_consensus.cc:686] T 7e1771f9c37e4a9bb8de117287615839 P 9e366e53c2474cd3ad5944198c93d7f8 [term 1 LEADER]: Becoming Leader. State: Replica: 9e366e53c2474cd3ad5944198c93d7f8, State: Running, Role: LEADER
I20260621 05:14:24.972334 23238 consensus_queue.cc:237] T 7e1771f9c37e4a9bb8de117287615839 P 9e366e53c2474cd3ad5944198c93d7f8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55344868c5da4ad3be557a66159f268c" member_type: VOTER last_known_addr { host: "127.3.131.130" port: 41983 } } peers { permanent_uuid: "9e366e53c2474cd3ad5944198c93d7f8" member_type: VOTER last_known_addr { host: "127.3.131.131" port: 46363 } } peers { permanent_uuid: "fe661fee975b4b2587e1f23dcdbddf9b" member_type: VOTER last_known_addr { host: "127.3.131.129" port: 39779 } }
I20260621 05:14:24.973461 22752 catalog_manager.cc:5495] T 7e1771f9c37e4a9bb8de117287615839 P 9e366e53c2474cd3ad5944198c93d7f8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9e366e53c2474cd3ad5944198c93d7f8 (127.3.131.131). New cstate: current_term: 1 leader_uuid: "9e366e53c2474cd3ad5944198c93d7f8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55344868c5da4ad3be557a66159f268c" member_type: VOTER last_known_addr { host: "127.3.131.130" port: 41983 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9e366e53c2474cd3ad5944198c93d7f8" member_type: VOTER last_known_addr { host: "127.3.131.131" port: 46363 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fe661fee975b4b2587e1f23dcdbddf9b" member_type: VOTER last_known_addr { host: "127.3.131.129" port: 39779 } health_report { overall_health: UNKNOWN } } }
I20260621 05:14:24.997742 23227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:14:24.995304 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40990 (local address 127.3.131.131:46363)
0621 05:14:24.995358 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0621 05:14:24.995360 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:14:24.995368 (+ 8us) server_negotiation.cc:407] Connection header received
0621 05:14:24.995402 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:14:24.995403 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:14:24.995420 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:14:24.995481 (+ 61us) server_negotiation.cc:226] Negotiated authn=TOKEN
0621 05:14:24.995604 (+ 123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:14:24.996103 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:14:24.996890 (+ 787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:14:24.997095 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:14:24.997159 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:14:24.997315 (+ 156us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0621 05:14:24.997429 (+ 114us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0621 05:14:24.997511 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0621 05:14:24.997595 (+ 84us) server_negotiation.cc:299] Negotiation successful
0621 05:14:24.997619 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260621 05:14:24.999673 23033 raft_consensus.cc:1261] T 7e1771f9c37e4a9bb8de117287615839 P 55344868c5da4ad3be557a66159f268c [term 1 FOLLOWER]: Refusing update from remote peer 9e366e53c2474cd3ad5944198c93d7f8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 05:14:24.999673 22899 raft_consensus.cc:1261] T 7e1771f9c37e4a9bb8de117287615839 P fe661fee975b4b2587e1f23dcdbddf9b [term 1 FOLLOWER]: Refusing update from remote peer 9e366e53c2474cd3ad5944198c93d7f8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 05:14:25.000064 23240 consensus_queue.cc:1035] T 7e1771f9c37e4a9bb8de117287615839 P 9e366e53c2474cd3ad5944198c93d7f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fe661fee975b4b2587e1f23dcdbddf9b" member_type: VOTER last_known_addr { host: "127.3.131.129" port: 39779 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 05:14:25.000200 23238 consensus_queue.cc:1035] T 7e1771f9c37e4a9bb8de117287615839 P 9e366e53c2474cd3ad5944198c93d7f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "55344868c5da4ad3be557a66159f268c" member_type: VOTER last_known_addr { host: "127.3.131.130" port: 41983 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 05:14:25.002846 23243 mvcc.cc:205] Tried to move back new op lower bound from 7299149271035678720 to 7299149270927339520. Current Snapshot: MvccSnapshot[applied={T|T < 7299149271035678720}]
I20260621 05:14:25.003484 23244 mvcc.cc:205] Tried to move back new op lower bound from 7299149271035678720 to 7299149270927339520. Current Snapshot: MvccSnapshot[applied={T|T < 7299149271035678720}]
I20260621 05:14:25.005261 22754 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37916:
table { table_name: "test-table" } modify_external_catalogs: true
I20260621 05:14:25.005367 22754 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37916:
table { table_name: "test-table" } modify_external_catalogs: true
I20260621 05:14:25.006283 22754 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 0c7f8bbeafb44227b9a2d125fc4ad530: Sending DeleteTablet for 3 replicas of tablet 7e1771f9c37e4a9bb8de117287615839
I20260621 05:14:25.006604 23013 tablet_service.cc:1514] Processing DeleteTablet for tablet 7e1771f9c37e4a9bb8de117287615839 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 05:14:24 UTC) from {username='kudu', principal='kudu/127.3.131.190@KRBTEST.COM'} at 127.0.0.1:48752
I20260621 05:14:25.006681 22879 tablet_service.cc:1514] Processing DeleteTablet for tablet 7e1771f9c37e4a9bb8de117287615839 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 05:14:24 UTC) from {username='kudu', principal='kudu/127.3.131.190@KRBTEST.COM'} at 127.0.0.1:49078
I20260621 05:14:25.006825 23147 tablet_service.cc:1514] Processing DeleteTablet for tablet 7e1771f9c37e4a9bb8de117287615839 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 05:14:24 UTC) from {username='kudu', principal='kudu/127.3.131.190@KRBTEST.COM'} at 127.0.0.1:40982
I20260621 05:14:25.006918 23252 tablet_replica.cc:330] T 7e1771f9c37e4a9bb8de117287615839 P 55344868c5da4ad3be557a66159f268c: stopping tablet replica
I20260621 05:14:25.007019 23252 raft_consensus.cc:2229] T 7e1771f9c37e4a9bb8de117287615839 P 55344868c5da4ad3be557a66159f268c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260621 05:14:25.007050 23254 tablet_replica.cc:330] T 7e1771f9c37e4a9bb8de117287615839 P 9e366e53c2474cd3ad5944198c93d7f8: stopping tablet replica
I20260621 05:14:25.007124 23252 raft_consensus.cc:2258] T 7e1771f9c37e4a9bb8de117287615839 P 55344868c5da4ad3be557a66159f268c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 05:14:25.007135 23254 raft_consensus.cc:2229] T 7e1771f9c37e4a9bb8de117287615839 P 9e366e53c2474cd3ad5944198c93d7f8 [term 1 LEADER]: Raft consensus shutting down.
I20260621 05:14:25.007252 23254 raft_consensus.cc:2258] T 7e1771f9c37e4a9bb8de117287615839 P 9e366e53c2474cd3ad5944198c93d7f8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 05:14:25.007316 23253 tablet_replica.cc:330] T 7e1771f9c37e4a9bb8de117287615839 P fe661fee975b4b2587e1f23dcdbddf9b: stopping tablet replica
I20260621 05:14:25.007503 23253 raft_consensus.cc:2229] T 7e1771f9c37e4a9bb8de117287615839 P fe661fee975b4b2587e1f23dcdbddf9b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260621 05:14:25.007524 3598 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWZOx2_/build/release/bin/kudu with pid 22816
I20260621 05:14:25.007616 23253 raft_consensus.cc:2258] T 7e1771f9c37e4a9bb8de117287615839 P fe661fee975b4b2587e1f23dcdbddf9b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 05:14:25.007700 23254 ts_tablet_manager.cc:1905] T 7e1771f9c37e4a9bb8de117287615839 P 9e366e53c2474cd3ad5944198c93d7f8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 05:14:25.008394 23252 ts_tablet_manager.cc:1905] T 7e1771f9c37e4a9bb8de117287615839 P 55344868c5da4ad3be557a66159f268c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 05:14:25.009723 23254 ts_tablet_manager.cc:1918] T 7e1771f9c37e4a9bb8de117287615839 P 9e366e53c2474cd3ad5944198c93d7f8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260621 05:14:25.009786 23254 log.cc:1196] T 7e1771f9c37e4a9bb8de117287615839 P 9e366e53c2474cd3ad5944198c93d7f8: Deleting WAL directory at /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782018616648578-3598-0/minicluster-data/ts-2/wal/wals/7e1771f9c37e4a9bb8de117287615839
I20260621 05:14:25.010111 23254 ts_tablet_manager.cc:1939] T 7e1771f9c37e4a9bb8de117287615839 P 9e366e53c2474cd3ad5944198c93d7f8: Deleting consensus metadata
I20260621 05:14:25.010429 23252 ts_tablet_manager.cc:1918] T 7e1771f9c37e4a9bb8de117287615839 P 55344868c5da4ad3be557a66159f268c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260621 05:14:25.010493 23252 log.cc:1196] T 7e1771f9c37e4a9bb8de117287615839 P 55344868c5da4ad3be557a66159f268c: Deleting WAL directory at /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782018616648578-3598-0/minicluster-data/ts-1/wal/wals/7e1771f9c37e4a9bb8de117287615839
I20260621 05:14:25.010771 23252 ts_tablet_manager.cc:1939] T 7e1771f9c37e4a9bb8de117287615839 P 55344868c5da4ad3be557a66159f268c: Deleting consensus metadata
I20260621 05:14:25.011206 22741 catalog_manager.cc:4841] TS 55344868c5da4ad3be557a66159f268c (127.3.131.130:41983): tablet 7e1771f9c37e4a9bb8de117287615839 (table test-table [id=6ee86190165c4ea5825860e59b19ef3c]) successfully deleted
I20260621 05:14:25.014588 3598 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWZOx2_/build/release/bin/kudu with pid 22950
W20260621 05:14:25.014667 22741 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.3.131.129:39779 (error 108)
I20260621 05:14:25.015143 22741 catalog_manager.cc:4841] TS 9e366e53c2474cd3ad5944198c93d7f8 (127.3.131.131:46363): tablet 7e1771f9c37e4a9bb8de117287615839 (table test-table [id=6ee86190165c4ea5825860e59b19ef3c]) successfully deleted
I20260621 05:14:25.016814 23224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:14:25.016674 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.131.129:39779 (local address 127.0.0.1:49086)
0621 05:14:25.016736 (+ 62us) negotiation.cc:107] Waiting for socket to connect
0621 05:14:25.016780 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.131.129:39779: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260621 05:14:25.016934 22741 catalog_manager.cc:4584] TS fe661fee975b4b2587e1f23dcdbddf9b (127.3.131.129:39779): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 7e1771f9c37e4a9bb8de117287615839: Network error: Client connection negotiation failed: client connection to 127.3.131.129:39779: connect: Connection refused (error 111)
I20260621 05:14:25.019845 3598 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWZOx2_/build/release/bin/kudu with pid 23084
I20260621 05:14:25.024289 3598 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWZOx2_/build/release/bin/kudu with pid 22722
2026-06-21T05:14:25Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (2176 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-06-21T05:14:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-21T05:14:25Z Disabled control of system clock
I20260621 05:14:25.052904 3598 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWZOx2_/build/release/bin/kudu
/tmp/dist-test-taskWZOx2_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782018616648578-3598-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782018616648578-3598-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782018616648578-3598-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782018616648578-3598-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.131.190:45663
--webserver_interface=127.3.131.190
--webserver_port=0
--builtin_ntp_servers=127.3.131.148:35031
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.131.190:45663
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260621 05:14:25.135855 23259 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 05:14:25.136123 23259 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 05:14:25.136147 23259 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 05:14:25.137419 23259 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 05:14:25.137463 23259 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 05:14:25.137476 23259 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 05:14:25.137490 23259 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 05:14:25.137501 23259 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 05:14:25.138880 23259 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.131.148:35031
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782018616648578-3598-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782018616648578-3598-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.131.190:45663
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.131.190:45663
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782018616648578-3598-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.131.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23259
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782018616648578-3598-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 21 Jun 2026 04:16:16 UTC on ccycloud.quasar-xycaha.root.comops.site
build id 4457881
I20260621 05:14:25.139094 23259 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 05:14:25.139427 23259 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 05:14:25.144713 23264 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:14:25.144757 23265 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:14:25.144814 23259 server_base.cc:1034] running on GCE node
W20260621 05:14:25.144748 23267 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:14:25.145190 23259 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 05:14:25.145534 23259 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 05:14:25.146723 23259 hybrid_clock.cc:648] HybridClock initialized: now 1782018865146707 us; error 27 us; skew 500 ppm
I20260621 05:14:25.148056 23259 webserver.cc:460] Webserver started at http://127.3.131.190:40421/ using document root <none> and password file <none>
I20260621 05:14:25.148352 23259 fs_manager.cc:359] Metadata directory not provided
I20260621 05:14:25.148424 23259 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 05:14:25.148620 23259 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 05:14:25.149709 23259 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782018616648578-3598-0/minicluster-data/master-0/data/instance:
uuid: "9a269880f1924c1ba2c4619ff627567d"
format_stamp: "Formatted at 2026-06-21 05:14:25 on dist-test-slave-vrq1"
server_key: "7e97bd0f29daf31c183444b1ecedf907"
server_key_iv: "cbdf909aedce9f45f7a967ad2922f1be"
server_key_version: "encryptionkey@0"
I20260621 05:14:25.150075 23259 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782018616648578-3598-0/minicluster-data/master-0/wal/instance:
uuid: "9a269880f1924c1ba2c4619ff627567d"
format_stamp: "Formatted at 2026-06-21 05:14:25 on dist-test-slave-vrq1"
server_key: "7e97bd0f29daf31c183444b1ecedf907"
server_key_iv: "cbdf909aedce9f45f7a967ad2922f1be"
server_key_version: "encryptionkey@0"
I20260621 05:14:25.151535 23259 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260621 05:14:25.152421 23273 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 05:14:25.152598 23259 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260621 05:14:25.152697 23259 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782018616648578-3598-0/minicluster-data/master-0/data,/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782018616648578-3598-0/minicluster-data/master-0/wal
uuid: "9a269880f1924c1ba2c4619ff627567d"
format_stamp: "Formatted at 2026-06-21 05:14:25 on dist-test-slave-vrq1"
server_key: "7e97bd0f29daf31c183444b1ecedf907"
server_key_iv: "cbdf909aedce9f45f7a967ad2922f1be"
server_key_version: "encryptionkey@0"
I20260621 05:14:25.152782 23259 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782018616648578-3598-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782018616648578-3598-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782018616648578-3598-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 05:14:25.166733 23259 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 05:14:25.167112 23259 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 05:14:25.167326 23259 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 05:14:25.171356 23259 rpc_server.cc:307] RPC server started. Bound to: 127.3.131.190:45663
I20260621 05:14:25.171396 23325 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.131.190:45663 every 8 connection(s)
I20260621 05:14:25.171768 23259 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782018616648578-3598-0/minicluster-data/master-0/data/info.pb
I20260621 05:14:25.172730 23326 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 05:14:25.174924 23326 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9a269880f1924c1ba2c4619ff627567d: Bootstrap starting.
I20260621 05:14:25.175563 23326 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9a269880f1924c1ba2c4619ff627567d: Neither blocks nor log segments found. Creating new log.
I20260621 05:14:25.175808 23326 log.cc:824] T 00000000000000000000000000000000 P 9a269880f1924c1ba2c4619ff627567d: Log is configured to *not* fsync() on all Append() calls
I20260621 05:14:25.176486 23326 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9a269880f1924c1ba2c4619ff627567d: No bootstrap required, opened a new log
I20260621 05:14:25.177790 3598 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWZOx2_/build/release/bin/kudu as pid 23259
I20260621 05:14:25.177914 3598 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782018616648578-3598-0/minicluster-data/master-0/wal/instance
I20260621 05:14:25.178095 3598 external_mini_cluster.cc:1428] Setting key 54bd972503f0d936321e6e9bc6c7d32d
I20260621 05:14:25.178086 23326 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9a269880f1924c1ba2c4619ff627567d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a269880f1924c1ba2c4619ff627567d" member_type: VOTER last_known_addr { host: "127.3.131.190" port: 45663 } }
I20260621 05:14:25.178308 23326 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9a269880f1924c1ba2c4619ff627567d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 05:14:25.178351 23326 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9a269880f1924c1ba2c4619ff627567d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a269880f1924c1ba2c4619ff627567d, State: Initialized, Role: FOLLOWER
I20260621 05:14:25.178509 23326 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9a269880f1924c1ba2c4619ff627567d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a269880f1924c1ba2c4619ff627567d" member_type: VOTER last_known_addr { host: "127.3.131.190" port: 45663 } }
I20260621 05:14:25.178615 23326 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9a269880f1924c1ba2c4619ff627567d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260621 05:14:25.178661 23326 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9a269880f1924c1ba2c4619ff627567d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260621 05:14:25.178714 23326 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9a269880f1924c1ba2c4619ff627567d [term 0 FOLLOWER]: Advancing to term 1
I20260621 05:14:25.179497 23326 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9a269880f1924c1ba2c4619ff627567d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a269880f1924c1ba2c4619ff627567d" member_type: VOTER last_known_addr { host: "127.3.131.190" port: 45663 } }
I20260621 05:14:25.179607 23326 leader_election.cc:302] T 00000000000000000000000000000000 P 9a269880f1924c1ba2c4619ff627567d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9a269880f1924c1ba2c4619ff627567d; no voters:
I20260621 05:14:25.179778 23326 leader_election.cc:288] T 00000000000000000000000000000000 P 9a269880f1924c1ba2c4619ff627567d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260621 05:14:25.179859 23331 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9a269880f1924c1ba2c4619ff627567d [term 1 FOLLOWER]: Leader election won for term 1
I20260621 05:14:25.180048 23331 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9a269880f1924c1ba2c4619ff627567d [term 1 LEADER]: Becoming Leader. State: Replica: 9a269880f1924c1ba2c4619ff627567d, State: Running, Role: LEADER
I20260621 05:14:25.180058 23326 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9a269880f1924c1ba2c4619ff627567d [sys.catalog]: configured and running, proceeding with master startup.
I20260621 05:14:25.180160 23331 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9a269880f1924c1ba2c4619ff627567d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a269880f1924c1ba2c4619ff627567d" member_type: VOTER last_known_addr { host: "127.3.131.190" port: 45663 } }
I20260621 05:14:25.180493 23331 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9a269880f1924c1ba2c4619ff627567d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9a269880f1924c1ba2c4619ff627567d. Latest consensus state: current_term: 1 leader_uuid: "9a269880f1924c1ba2c4619ff627567d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a269880f1924c1ba2c4619ff627567d" member_type: VOTER last_known_addr { host: "127.3.131.190" port: 45663 } } }
I20260621 05:14:25.180585 23331 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9a269880f1924c1ba2c4619ff627567d [sys.catalog]: This master's current role is: LEADER
I20260621 05:14:25.180502 23332 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9a269880f1924c1ba2c4619ff627567d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9a269880f1924c1ba2c4619ff627567d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a269880f1924c1ba2c4619ff627567d" member_type: VOTER last_known_addr { host: "127.3.131.190" port: 45663 } } }
I20260621 05:14:25.180811 23338 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260621 05:14:25.180821 23332 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9a269880f1924c1ba2c4619ff627567d [sys.catalog]: This master's current role is: LEADER
I20260621 05:14:25.181352 23338 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260621 05:14:25.182958 23338 catalog_manager.cc:1347] Generated new cluster ID: 2bc93585eaf74e72baf427f45ffbb9a3
I20260621 05:14:25.183007 23338 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260621 05:14:25.184839 23330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:14:25.178581 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46134 (local address 127.3.131.190:45663)
0621 05:14:25.178832 (+ 251us) server_negotiation.cc:206] Beginning negotiation
0621 05:14:25.178841 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0621 05:14:25.178876 (+ 35us) server_negotiation.cc:407] Connection header received
0621 05:14:25.178993 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:14:25.179000 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:14:25.179082 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:14:25.179274 (+ 192us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:14:25.179504 (+ 230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:14:25.181513 (+ 2009us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:14:25.182744 (+ 1231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:14:25.183300 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:14:25.183427 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:14:25.184136 (+ 709us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:14:25.184140 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:14:25.184140 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 05:14:25.184316 (+ 176us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:14:25.184348 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 05:14:25.184359 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:14:25.184429 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0621 05:14:25.184504 (+ 75us) server_negotiation.cc:299] Negotiation successful
0621 05:14:25.184682 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":176,"thread_start_us":109,"threads_started":1}
I20260621 05:14:25.201110 23338 catalog_manager.cc:1370] Generated new certificate authority record
I20260621 05:14:25.201684 23338 catalog_manager.cc:1504] Loading token signing keys...
I20260621 05:14:25.207280 23338 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9a269880f1924c1ba2c4619ff627567d: Generated new TSK 0
I20260621 05:14:25.207551 23338 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260621 05:14:25.209546 3598 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWZOx2_/build/release/bin/kudu
/tmp/dist-test-taskWZOx2_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782018616648578-3598-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782018616648578-3598-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782018616648578-3598-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782018616648578-3598-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.131.129:0
--local_ip_for_outbound_sockets=127.3.131.129
--webserver_interface=127.3.131.129
--webserver_port=0
--tserver_master_addrs=127.3.131.190:45663
--builtin_ntp_servers=127.3.131.148:35031
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260621 05:14:25.305120 23350 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 05:14:25.305415 23350 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 05:14:25.305459 23350 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 05:14:25.306759 23350 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 05:14:25.306841 23350 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 05:14:25.306902 23350 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.131.129
I20260621 05:14:25.308478 23350 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.131.148:35031
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782018616648578-3598-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782018616648578-3598-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.131.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782018616648578-3598-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.131.129
--webserver_port=0
--tserver_master_addrs=127.3.131.190:45663
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23350
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.131.129
--log_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782018616648578-3598-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 21 Jun 2026 04:16:16 UTC on ccycloud.quasar-xycaha.root.comops.site
build id 4457881
I20260621 05:14:25.308794 23350 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 05:14:25.309130 23350 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 05:14:25.314790 23358 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:14:25.314783 23355 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:14:25.314815 23356 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:14:25.315493 23350 server_base.cc:1034] running on GCE node
I20260621 05:14:25.315774 23350 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 05:14:25.316069 23350 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 05:14:25.317238 23350 hybrid_clock.cc:648] HybridClock initialized: now 1782018865317215 us; error 35 us; skew 500 ppm
I20260621 05:14:25.318730 23350 webserver.cc:460] Webserver started at http://127.3.131.129:35071/ using document root <none> and password file <none>
I20260621 05:14:25.319017 23350 fs_manager.cc:359] Metadata directory not provided
I20260621 05:14:25.319084 23350 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 05:14:25.319288 23350 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 05:14:25.320407 23350 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782018616648578-3598-0/minicluster-data/ts-0/data/instance:
uuid: "777a8c1010a34e578bcb7861ab6cd9f8"
format_stamp: "Formatted at 2026-06-21 05:14:25 on dist-test-slave-vrq1"
server_key: "c39d1fd6db42ff5793e59b720f3c4e40"
server_key_iv: "d6e065a170d4f8537ed63af58a2c669d"
server_key_version: "encryptionkey@0"
I20260621 05:14:25.320799 23350 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782018616648578-3598-0/minicluster-data/ts-0/wal/instance:
uuid: "777a8c1010a34e578bcb7861ab6cd9f8"
format_stamp: "Formatted at 2026-06-21 05:14:25 on dist-test-slave-vrq1"
server_key: "c39d1fd6db42ff5793e59b720f3c4e40"
server_key_iv: "d6e065a170d4f8537ed63af58a2c669d"
server_key_version: "encryptionkey@0"
I20260621 05:14:25.322206 23350 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260621 05:14:25.322994 23364 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 05:14:25.323158 23350 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260621 05:14:25.323278 23350 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782018616648578-3598-0/minicluster-data/ts-0/data,/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782018616648578-3598-0/minicluster-data/ts-0/wal
uuid: "777a8c1010a34e578bcb7861ab6cd9f8"
format_stamp: "Formatted at 2026-06-21 05:14:25 on dist-test-slave-vrq1"
server_key: "c39d1fd6db42ff5793e59b720f3c4e40"
server_key_iv: "d6e065a170d4f8537ed63af58a2c669d"
server_key_version: "encryptionkey@0"
I20260621 05:14:25.323356 23350 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782018616648578-3598-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782018616648578-3598-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782018616648578-3598-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 05:14:25.348546 23350 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 05:14:25.348899 23350 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 05:14:25.349057 23350 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 05:14:25.349304 23350 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 05:14:25.349654 23350 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 05:14:25.349690 23350 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 05:14:25.349713 23350 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 05:14:25.349735 23350 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 05:14:25.358031 23350 rpc_server.cc:307] RPC server started. Bound to: 127.3.131.129:39835
I20260621 05:14:25.358156 23477 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.131.129:39835 every 8 connection(s)
I20260621 05:14:25.358470 23350 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782018616648578-3598-0/minicluster-data/ts-0/data/info.pb
I20260621 05:14:25.364070 23330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:14:25.359326 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.131.129:50553 (local address 127.3.131.190:45663)
0621 05:14:25.359374 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0621 05:14:25.359376 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:14:25.359606 (+ 230us) server_negotiation.cc:407] Connection header received
0621 05:14:25.360128 (+ 522us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:14:25.360130 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:14:25.360146 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:14:25.360204 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:14:25.360878 (+ 674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:14:25.361381 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:14:25.362438 (+ 1057us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:14:25.362644 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:14:25.362702 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:14:25.363723 (+ 1021us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:14:25.363725 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:14:25.363725 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 05:14:25.363729 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:14:25.363743 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 05:14:25.363747 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:14:25.363792 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0621 05:14:25.363977 (+ 185us) server_negotiation.cc:299] Negotiation successful
0621 05:14:25.364006 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260621 05:14:25.364279 23480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:14:25.359291 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.131.190:45663 (local address 127.3.131.129:50553)
0621 05:14:25.359471 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0621 05:14:25.359506 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0621 05:14:25.360008 (+ 502us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:14:25.360257 (+ 249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:14:25.360260 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:14:25.360268 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:14:25.360784 (+ 516us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:14:25.360788 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:14:25.361483 (+ 695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:14:25.361488 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:14:25.362346 (+ 858us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:14:25.362349 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:14:25.362771 (+ 422us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:14:25.362772 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:14:25.362892 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:14:25.363574 (+ 682us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0621 05:14:25.363574 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:14:25.363605 (+ 31us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0621 05:14:25.363628 (+ 23us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 05:14:25.363850 (+ 222us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 05:14:25.363857 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 05:14:25.363860 (+ 3us) client_negotiation.cc:769] Sending connection context
0621 05:14:25.363934 (+ 74us) client_negotiation.cc:240] Negotiation successful
0621 05:14:25.364034 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":88,"threads_started":1}
I20260621 05:14:25.364938 23478 heartbeater.cc:347] Connected to a master server at 127.3.131.190:45663
I20260621 05:14:25.365041 23478 heartbeater.cc:464] Registering TS with master...
I20260621 05:14:25.365195 23478 heartbeater.cc:511] Master 127.3.131.190:45663 requested a full tablet report, sending...
I20260621 05:14:25.365741 23290 ts_manager.cc:194] Registered new tserver with Master: 777a8c1010a34e578bcb7861ab6cd9f8 (127.3.131.129:39835)
I20260621 05:14:25.366189 3598 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWZOx2_/build/release/bin/kudu as pid 23350
I20260621 05:14:25.366299 3598 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782018616648578-3598-0/minicluster-data/ts-0/wal/instance
I20260621 05:14:25.366506 3598 external_mini_cluster.cc:1428] Setting key e9b735fcf168d57db9cfb1582516646a
I20260621 05:14:25.366813 23290 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.3.131.129:50553
I20260621 05:14:25.367812 3598 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWZOx2_/build/release/bin/kudu
/tmp/dist-test-taskWZOx2_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782018616648578-3598-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782018616648578-3598-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782018616648578-3598-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782018616648578-3598-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.131.130:0
--local_ip_for_outbound_sockets=127.3.131.130
--webserver_interface=127.3.131.130
--webserver_port=0
--tserver_master_addrs=127.3.131.190:45663
--builtin_ntp_servers=127.3.131.148:35031
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260621 05:14:25.453663 23481 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 05:14:25.453873 23481 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 05:14:25.453896 23481 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 05:14:25.455145 23481 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 05:14:25.455257 23481 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 05:14:25.455307 23481 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.131.130
I20260621 05:14:25.456842 23481 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.131.148:35031
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782018616648578-3598-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782018616648578-3598-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.131.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782018616648578-3598-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.131.130
--webserver_port=0
--tserver_master_addrs=127.3.131.190:45663
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23481
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.131.130
--log_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782018616648578-3598-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 21 Jun 2026 04:16:16 UTC on ccycloud.quasar-xycaha.root.comops.site
build id 4457881
I20260621 05:14:25.457201 23481 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 05:14:25.457507 23481 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 05:14:25.462913 23486 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:14:25.462916 23489 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:14:25.462970 23487 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:14:25.463282 23481 server_base.cc:1034] running on GCE node
I20260621 05:14:25.463548 23481 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 05:14:25.463810 23481 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 05:14:25.465018 23481 hybrid_clock.cc:648] HybridClock initialized: now 1782018865465009 us; error 34 us; skew 500 ppm
I20260621 05:14:25.466450 23481 webserver.cc:460] Webserver started at http://127.3.131.130:39371/ using document root <none> and password file <none>
I20260621 05:14:25.466728 23481 fs_manager.cc:359] Metadata directory not provided
I20260621 05:14:25.466804 23481 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 05:14:25.466962 23481 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 05:14:25.468163 23481 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782018616648578-3598-0/minicluster-data/ts-1/data/instance:
uuid: "bdef0b77b17f45ffa137ffedc1b131fe"
format_stamp: "Formatted at 2026-06-21 05:14:25 on dist-test-slave-vrq1"
server_key: "4abc7152f20214655f3add70147a40b5"
server_key_iv: "672427a3bd85d3382332991eb28b2c4d"
server_key_version: "encryptionkey@0"
I20260621 05:14:25.468541 23481 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782018616648578-3598-0/minicluster-data/ts-1/wal/instance:
uuid: "bdef0b77b17f45ffa137ffedc1b131fe"
format_stamp: "Formatted at 2026-06-21 05:14:25 on dist-test-slave-vrq1"
server_key: "4abc7152f20214655f3add70147a40b5"
server_key_iv: "672427a3bd85d3382332991eb28b2c4d"
server_key_version: "encryptionkey@0"
I20260621 05:14:25.469946 23481 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260621 05:14:25.470959 23495 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 05:14:25.471167 23481 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260621 05:14:25.471302 23481 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782018616648578-3598-0/minicluster-data/ts-1/data,/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782018616648578-3598-0/minicluster-data/ts-1/wal
uuid: "bdef0b77b17f45ffa137ffedc1b131fe"
format_stamp: "Formatted at 2026-06-21 05:14:25 on dist-test-slave-vrq1"
server_key: "4abc7152f20214655f3add70147a40b5"
server_key_iv: "672427a3bd85d3382332991eb28b2c4d"
server_key_version: "encryptionkey@0"
I20260621 05:14:25.471391 23481 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782018616648578-3598-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782018616648578-3598-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782018616648578-3598-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 05:14:25.500423 23481 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 05:14:25.500769 23481 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 05:14:25.500919 23481 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 05:14:25.501173 23481 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 05:14:25.501511 23481 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 05:14:25.501564 23481 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 05:14:25.501603 23481 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 05:14:25.501639 23481 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 05:14:25.508186 23481 rpc_server.cc:307] RPC server started. Bound to: 127.3.131.130:45893
I20260621 05:14:25.508234 23608 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.131.130:45893 every 8 connection(s)
I20260621 05:14:25.508595 23481 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782018616648578-3598-0/minicluster-data/ts-1/data/info.pb
I20260621 05:14:25.513965 3598 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWZOx2_/build/release/bin/kudu as pid 23481
I20260621 05:14:25.514113 3598 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782018616648578-3598-0/minicluster-data/ts-1/wal/instance
I20260621 05:14:25.514357 3598 external_mini_cluster.cc:1428] Setting key 60965b78d8283e4f7510f75a3e506a9f
I20260621 05:14:25.515021 23330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:14:25.509474 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.131.130:54491 (local address 127.3.131.190:45663)
0621 05:14:25.509595 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0621 05:14:25.509598 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0621 05:14:25.509963 (+ 365us) server_negotiation.cc:407] Connection header received
0621 05:14:25.510525 (+ 562us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:14:25.510526 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:14:25.510538 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:14:25.510593 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:14:25.511423 (+ 830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:14:25.511938 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:14:25.513144 (+ 1206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:14:25.513419 (+ 275us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:14:25.513522 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:14:25.514736 (+ 1214us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:14:25.514737 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:14:25.514737 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 05:14:25.514743 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:14:25.514759 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 05:14:25.514763 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:14:25.514811 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0621 05:14:25.514889 (+ 78us) server_negotiation.cc:299] Negotiation successful
0621 05:14:25.514923 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260621 05:14:25.515297 23611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:14:25.509366 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.131.190:45663 (local address 127.3.131.130:54491)
0621 05:14:25.509653 (+ 287us) negotiation.cc:107] Waiting for socket to connect
0621 05:14:25.509825 (+ 172us) client_negotiation.cc:174] Beginning negotiation
0621 05:14:25.510408 (+ 583us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:14:25.510719 (+ 311us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:14:25.510723 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:14:25.510731 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:14:25.511326 (+ 595us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:14:25.511331 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:14:25.512041 (+ 710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:14:25.512042 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:14:25.513023 (+ 981us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:14:25.513028 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:14:25.513554 (+ 526us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:14:25.513555 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:14:25.513771 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:14:25.514563 (+ 792us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0621 05:14:25.514564 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:14:25.514612 (+ 48us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0621 05:14:25.514648 (+ 36us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 05:14:25.514813 (+ 165us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 05:14:25.514818 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 05:14:25.514822 (+ 4us) client_negotiation.cc:769] Sending connection context
0621 05:14:25.514885 (+ 63us) client_negotiation.cc:240] Negotiation successful
0621 05:14:25.515001 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":216,"thread_start_us":109,"threads_started":1}
I20260621 05:14:25.515700 3598 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWZOx2_/build/release/bin/kudu
/tmp/dist-test-taskWZOx2_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782018616648578-3598-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782018616648578-3598-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782018616648578-3598-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782018616648578-3598-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.131.131:0
--local_ip_for_outbound_sockets=127.3.131.131
--webserver_interface=127.3.131.131
--webserver_port=0
--tserver_master_addrs=127.3.131.190:45663
--builtin_ntp_servers=127.3.131.148:35031
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260621 05:14:25.516031 23609 heartbeater.cc:347] Connected to a master server at 127.3.131.190:45663
I20260621 05:14:25.516130 23609 heartbeater.cc:464] Registering TS with master...
I20260621 05:14:25.516358 23609 heartbeater.cc:511] Master 127.3.131.190:45663 requested a full tablet report, sending...
I20260621 05:14:25.516870 23289 ts_manager.cc:194] Registered new tserver with Master: bdef0b77b17f45ffa137ffedc1b131fe (127.3.131.130:45893)
I20260621 05:14:25.517410 23289 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.3.131.130:54491
W20260621 05:14:25.600509 23612 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 05:14:25.600759 23612 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 05:14:25.600785 23612 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 05:14:25.602077 23612 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 05:14:25.602128 23612 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 05:14:25.602155 23612 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.131.131
I20260621 05:14:25.603713 23612 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.131.148:35031
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782018616648578-3598-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782018616648578-3598-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.131.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782018616648578-3598-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.131.131
--webserver_port=0
--tserver_master_addrs=127.3.131.190:45663
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23612
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.131.131
--log_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782018616648578-3598-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 21 Jun 2026 04:16:16 UTC on ccycloud.quasar-xycaha.root.comops.site
build id 4457881
I20260621 05:14:25.603983 23612 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 05:14:25.604310 23612 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 05:14:25.610378 23618 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:14:25.610361 23617 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:14:25.610361 23620 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:14:25.610986 23612 server_base.cc:1034] running on GCE node
I20260621 05:14:25.611488 23612 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 05:14:25.611807 23612 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 05:14:25.613004 23612 hybrid_clock.cc:648] HybridClock initialized: now 1782018865612983 us; error 46 us; skew 500 ppm
I20260621 05:14:25.614491 23612 webserver.cc:460] Webserver started at http://127.3.131.131:33447/ using document root <none> and password file <none>
I20260621 05:14:25.614822 23612 fs_manager.cc:359] Metadata directory not provided
I20260621 05:14:25.614903 23612 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 05:14:25.615074 23612 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 05:14:25.616276 23612 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782018616648578-3598-0/minicluster-data/ts-2/data/instance:
uuid: "739f2ec1059e42e0824f47dbf1912308"
format_stamp: "Formatted at 2026-06-21 05:14:25 on dist-test-slave-vrq1"
server_key: "6ef8307cfba61cf8f63ee5225ad19c1b"
server_key_iv: "f3420d7a63692f451b02d62892408421"
server_key_version: "encryptionkey@0"
I20260621 05:14:25.616689 23612 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782018616648578-3598-0/minicluster-data/ts-2/wal/instance:
uuid: "739f2ec1059e42e0824f47dbf1912308"
format_stamp: "Formatted at 2026-06-21 05:14:25 on dist-test-slave-vrq1"
server_key: "6ef8307cfba61cf8f63ee5225ad19c1b"
server_key_iv: "f3420d7a63692f451b02d62892408421"
server_key_version: "encryptionkey@0"
I20260621 05:14:25.618623 23612 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.003s
I20260621 05:14:25.619573 23626 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 05:14:25.619814 23612 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260621 05:14:25.619930 23612 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782018616648578-3598-0/minicluster-data/ts-2/data,/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782018616648578-3598-0/minicluster-data/ts-2/wal
uuid: "739f2ec1059e42e0824f47dbf1912308"
format_stamp: "Formatted at 2026-06-21 05:14:25 on dist-test-slave-vrq1"
server_key: "6ef8307cfba61cf8f63ee5225ad19c1b"
server_key_iv: "f3420d7a63692f451b02d62892408421"
server_key_version: "encryptionkey@0"
I20260621 05:14:25.620026 23612 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782018616648578-3598-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782018616648578-3598-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782018616648578-3598-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 05:14:25.640033 23612 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 05:14:25.640446 23612 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 05:14:25.640622 23612 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 05:14:25.640900 23612 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 05:14:25.641301 23612 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 05:14:25.641361 23612 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 05:14:25.641398 23612 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 05:14:25.641441 23612 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 05:14:25.647933 23612 rpc_server.cc:307] RPC server started. Bound to: 127.3.131.131:34855
I20260621 05:14:25.648000 23739 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.131.131:34855 every 8 connection(s)
I20260621 05:14:25.648416 23612 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782018616648578-3598-0/minicluster-data/ts-2/data/info.pb
I20260621 05:14:25.652146 3598 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWZOx2_/build/release/bin/kudu as pid 23612
I20260621 05:14:25.652249 3598 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782018616648578-3598-0/minicluster-data/ts-2/wal/instance
I20260621 05:14:25.652482 3598 external_mini_cluster.cc:1428] Setting key 44d21a56d18c36d2dc14cf0870fbb631
I20260621 05:14:25.654810 23330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:14:25.649213 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.131.131:43961 (local address 127.3.131.190:45663)
0621 05:14:25.649287 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0621 05:14:25.649290 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0621 05:14:25.649551 (+ 261us) server_negotiation.cc:407] Connection header received
0621 05:14:25.650181 (+ 630us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:14:25.650182 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:14:25.650193 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:14:25.650261 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:14:25.650993 (+ 732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:14:25.651692 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:14:25.652936 (+ 1244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:14:25.653306 (+ 370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:14:25.653382 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:14:25.654460 (+ 1078us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:14:25.654461 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:14:25.654461 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 05:14:25.654466 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:14:25.654482 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 05:14:25.654486 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:14:25.654543 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0621 05:14:25.654711 (+ 168us) server_negotiation.cc:299] Negotiation successful
0621 05:14:25.654745 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260621 05:14:25.655006 23742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:14:25.649204 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.131.190:45663 (local address 127.3.131.131:43961)
0621 05:14:25.649419 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0621 05:14:25.649456 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0621 05:14:25.650026 (+ 570us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:14:25.650338 (+ 312us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:14:25.650342 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:14:25.650351 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:14:25.650877 (+ 526us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:14:25.650881 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:14:25.651798 (+ 917us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:14:25.651799 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:14:25.652854 (+ 1055us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:14:25.652857 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:14:25.653420 (+ 563us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:14:25.653421 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:14:25.653601 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:14:25.654269 (+ 668us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0621 05:14:25.654269 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:14:25.654309 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0621 05:14:25.654340 (+ 31us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 05:14:25.654554 (+ 214us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 05:14:25.654559 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 05:14:25.654562 (+ 3us) client_negotiation.cc:769] Sending connection context
0621 05:14:25.654664 (+ 102us) client_negotiation.cc:240] Negotiation successful
0621 05:14:25.654767 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":152,"thread_start_us":109,"threads_started":1}
I20260621 05:14:25.655563 23740 heartbeater.cc:347] Connected to a master server at 127.3.131.190:45663
I20260621 05:14:25.655725 23740 heartbeater.cc:464] Registering TS with master...
I20260621 05:14:25.655930 23740 heartbeater.cc:511] Master 127.3.131.190:45663 requested a full tablet report, sending...
I20260621 05:14:25.656353 23289 ts_manager.cc:194] Registered new tserver with Master: 739f2ec1059e42e0824f47dbf1912308 (127.3.131.131:34855)
I20260621 05:14:25.656754 23289 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.3.131.131:43961
I20260621 05:14:25.665082 3598 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260621 05:14:25.669978 23330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:14:25.666740 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46140 (local address 127.3.131.190:45663)
0621 05:14:25.666816 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0621 05:14:25.666818 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:14:25.666911 (+ 93us) server_negotiation.cc:407] Connection header received
0621 05:14:25.666968 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:14:25.666969 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:14:25.666981 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:14:25.667028 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:14:25.667334 (+ 306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:14:25.667824 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:14:25.668640 (+ 816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:14:25.668869 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:14:25.668940 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:14:25.669636 (+ 696us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:14:25.669637 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:14:25.669637 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 05:14:25.669642 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:14:25.669655 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 05:14:25.669659 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:14:25.669707 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0621 05:14:25.669825 (+ 118us) server_negotiation.cc:299] Negotiation successful
0621 05:14:25.669851 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260621 05:14:25.672268 23289 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46140:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260621 05:14:25.674168 23289 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260621 05:14:25.681972 23752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:14:25.677287 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37380 (local address 127.3.131.130:45893)
0621 05:14:25.677486 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0621 05:14:25.677491 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 05:14:25.677772 (+ 281us) server_negotiation.cc:407] Connection header received
0621 05:14:25.677821 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:14:25.677823 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:14:25.677839 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:14:25.677927 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 05:14:25.678295 (+ 368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:14:25.679286 (+ 991us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:14:25.680622 (+ 1336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:14:25.681523 (+ 901us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:14:25.681604 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:14:25.681706 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0621 05:14:25.681839 (+ 133us) server_negotiation.cc:299] Negotiation successful
0621 05:14:25.681881 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":168,"thread_start_us":95,"threads_started":1}
I20260621 05:14:25.682053 23749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:14:25.677044 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.131.130:45893 (local address 127.0.0.1:37380)
0621 05:14:25.677478 (+ 434us) negotiation.cc:107] Waiting for socket to connect
0621 05:14:25.677529 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0621 05:14:25.677657 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:14:25.678077 (+ 420us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:14:25.678083 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:14:25.678088 (+ 5us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 05:14:25.678206 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:14:25.678209 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:14:25.679369 (+ 1160us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:14:25.679370 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:14:25.680496 (+ 1126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:14:25.680499 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:14:25.681624 (+ 1125us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:14:25.681625 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:14:25.681715 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:14:25.681731 (+ 16us) client_negotiation.cc:769] Sending connection context
0621 05:14:25.681811 (+ 80us) client_negotiation.cc:240] Negotiation successful
0621 05:14:25.681869 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":399,"thread_start_us":86,"threads_started":1}
I20260621 05:14:25.682740 23751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:14:25.677260 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43128 (local address 127.3.131.129:39835)
0621 05:14:25.677460 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0621 05:14:25.677468 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0621 05:14:25.678364 (+ 896us) server_negotiation.cc:407] Connection header received
0621 05:14:25.678408 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:14:25.678411 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:14:25.678426 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:14:25.678495 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 05:14:25.678680 (+ 185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:14:25.679847 (+ 1167us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:14:25.681309 (+ 1462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:14:25.682219 (+ 910us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:14:25.682293 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:14:25.682386 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0621 05:14:25.682604 (+ 218us) server_negotiation.cc:299] Negotiation successful
0621 05:14:25.682652 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":152,"thread_start_us":107,"threads_started":1}
I20260621 05:14:25.682926 23753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:14:25.677396 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51408 (local address 127.3.131.131:34855)
0621 05:14:25.677618 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0621 05:14:25.677623 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 05:14:25.678335 (+ 712us) server_negotiation.cc:407] Connection header received
0621 05:14:25.678394 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:14:25.678396 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:14:25.678409 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:14:25.678499 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 05:14:25.678615 (+ 116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:14:25.679810 (+ 1195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:14:25.681362 (+ 1552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:14:25.682327 (+ 965us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:14:25.682397 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:14:25.682483 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0621 05:14:25.682797 (+ 314us) server_negotiation.cc:299] Negotiation successful
0621 05:14:25.682843 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":186,"thread_start_us":88,"threads_started":1}
I20260621 05:14:25.682951 23750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:14:25.677060 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.131.131:34855 (local address 127.0.0.1:51408)
0621 05:14:25.678232 (+ 1172us) negotiation.cc:107] Waiting for socket to connect
0621 05:14:25.678240 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0621 05:14:25.678303 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:14:25.678499 (+ 196us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:14:25.678499 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:14:25.678502 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 05:14:25.678556 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:14:25.678558 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:14:25.679938 (+ 1380us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:14:25.679939 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:14:25.681293 (+ 1354us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:14:25.681295 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:14:25.682634 (+ 1339us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:14:25.682635 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:14:25.682690 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:14:25.682705 (+ 15us) client_negotiation.cc:769] Sending connection context
0621 05:14:25.682758 (+ 53us) client_negotiation.cc:240] Negotiation successful
0621 05:14:25.682815 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1162,"mutex_wait_us":117,"thread_start_us":68,"threads_started":1}
I20260621 05:14:25.683161 23748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:14:25.677044 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.131.129:39835 (local address 127.0.0.1:43128)
0621 05:14:25.678095 (+ 1051us) negotiation.cc:107] Waiting for socket to connect
0621 05:14:25.678105 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0621 05:14:25.678160 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:14:25.678566 (+ 406us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:14:25.678566 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:14:25.678568 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 05:14:25.678613 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:14:25.678615 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:14:25.679935 (+ 1320us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:14:25.679936 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:14:25.681203 (+ 1267us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:14:25.681206 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:14:25.682352 (+ 1146us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:14:25.682353 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:14:25.682421 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:14:25.682426 (+ 5us) client_negotiation.cc:769] Sending connection context
0621 05:14:25.682473 (+ 47us) client_negotiation.cc:240] Negotiation successful
0621 05:14:25.682966 (+ 493us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1030,"mutex_wait_us":47,"thread_start_us":94,"threads_started":1}
I20260621 05:14:25.683344 23543 tablet_service.cc:1467] Processing CreateTablet for tablet 6506986b8cee49ecac503df598aec03c (DEFAULT_TABLE table=test-table [id=88cefc4f787543e899fa5579200dd020]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 05:14:25.683593 23674 tablet_service.cc:1467] Processing CreateTablet for tablet 6506986b8cee49ecac503df598aec03c (DEFAULT_TABLE table=test-table [id=88cefc4f787543e899fa5579200dd020]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 05:14:25.683765 23543 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6506986b8cee49ecac503df598aec03c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 05:14:25.683892 23674 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6506986b8cee49ecac503df598aec03c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 05:14:25.687427 23754 tablet_bootstrap.cc:492] T 6506986b8cee49ecac503df598aec03c P bdef0b77b17f45ffa137ffedc1b131fe: Bootstrap starting.
I20260621 05:14:25.687811 23755 tablet_bootstrap.cc:492] T 6506986b8cee49ecac503df598aec03c P 739f2ec1059e42e0824f47dbf1912308: Bootstrap starting.
I20260621 05:14:25.688162 23412 tablet_service.cc:1467] Processing CreateTablet for tablet 6506986b8cee49ecac503df598aec03c (DEFAULT_TABLE table=test-table [id=88cefc4f787543e899fa5579200dd020]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 05:14:25.688519 23412 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6506986b8cee49ecac503df598aec03c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 05:14:25.688560 23755 tablet_bootstrap.cc:654] T 6506986b8cee49ecac503df598aec03c P 739f2ec1059e42e0824f47dbf1912308: Neither blocks nor log segments found. Creating new log.
I20260621 05:14:25.688887 23755 log.cc:824] T 6506986b8cee49ecac503df598aec03c P 739f2ec1059e42e0824f47dbf1912308: Log is configured to *not* fsync() on all Append() calls
I20260621 05:14:25.689729 23755 tablet_bootstrap.cc:492] T 6506986b8cee49ecac503df598aec03c P 739f2ec1059e42e0824f47dbf1912308: No bootstrap required, opened a new log
I20260621 05:14:25.689766 23754 tablet_bootstrap.cc:654] T 6506986b8cee49ecac503df598aec03c P bdef0b77b17f45ffa137ffedc1b131fe: Neither blocks nor log segments found. Creating new log.
I20260621 05:14:25.689836 23755 ts_tablet_manager.cc:1397] T 6506986b8cee49ecac503df598aec03c P 739f2ec1059e42e0824f47dbf1912308: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260621 05:14:25.690089 23754 log.cc:824] T 6506986b8cee49ecac503df598aec03c P bdef0b77b17f45ffa137ffedc1b131fe: Log is configured to *not* fsync() on all Append() calls
I20260621 05:14:25.690985 23754 tablet_bootstrap.cc:492] T 6506986b8cee49ecac503df598aec03c P bdef0b77b17f45ffa137ffedc1b131fe: No bootstrap required, opened a new log
I20260621 05:14:25.691062 23754 ts_tablet_manager.cc:1397] T 6506986b8cee49ecac503df598aec03c P bdef0b77b17f45ffa137ffedc1b131fe: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20260621 05:14:25.692373 23758 tablet_bootstrap.cc:492] T 6506986b8cee49ecac503df598aec03c P 777a8c1010a34e578bcb7861ab6cd9f8: Bootstrap starting.
I20260621 05:14:25.692299 23755 raft_consensus.cc:348] T 6506986b8cee49ecac503df598aec03c P 739f2ec1059e42e0824f47dbf1912308 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "777a8c1010a34e578bcb7861ab6cd9f8" member_type: VOTER last_known_addr { host: "127.3.131.129" port: 39835 } } peers { permanent_uuid: "bdef0b77b17f45ffa137ffedc1b131fe" member_type: VOTER last_known_addr { host: "127.3.131.130" port: 45893 } } peers { permanent_uuid: "739f2ec1059e42e0824f47dbf1912308" member_type: VOTER last_known_addr { host: "127.3.131.131" port: 34855 } }
I20260621 05:14:25.692485 23755 raft_consensus.cc:374] T 6506986b8cee49ecac503df598aec03c P 739f2ec1059e42e0824f47dbf1912308 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 05:14:25.692528 23755 raft_consensus.cc:729] T 6506986b8cee49ecac503df598aec03c P 739f2ec1059e42e0824f47dbf1912308 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 739f2ec1059e42e0824f47dbf1912308, State: Initialized, Role: FOLLOWER
I20260621 05:14:25.692667 23755 consensus_queue.cc:260] T 6506986b8cee49ecac503df598aec03c P 739f2ec1059e42e0824f47dbf1912308 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "777a8c1010a34e578bcb7861ab6cd9f8" member_type: VOTER last_known_addr { host: "127.3.131.129" port: 39835 } } peers { permanent_uuid: "bdef0b77b17f45ffa137ffedc1b131fe" member_type: VOTER last_known_addr { host: "127.3.131.130" port: 45893 } } peers { permanent_uuid: "739f2ec1059e42e0824f47dbf1912308" member_type: VOTER last_known_addr { host: "127.3.131.131" port: 34855 } }
I20260621 05:14:25.692970 23755 ts_tablet_manager.cc:1428] T 6506986b8cee49ecac503df598aec03c P 739f2ec1059e42e0824f47dbf1912308: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260621 05:14:25.693037 23740 heartbeater.cc:503] Master 127.3.131.190:45663 was elected leader, sending a full tablet report...
I20260621 05:14:25.693257 23758 tablet_bootstrap.cc:654] T 6506986b8cee49ecac503df598aec03c P 777a8c1010a34e578bcb7861ab6cd9f8: Neither blocks nor log segments found. Creating new log.
I20260621 05:14:25.693629 23758 log.cc:824] T 6506986b8cee49ecac503df598aec03c P 777a8c1010a34e578bcb7861ab6cd9f8: Log is configured to *not* fsync() on all Append() calls
I20260621 05:14:25.693925 23754 raft_consensus.cc:348] T 6506986b8cee49ecac503df598aec03c P bdef0b77b17f45ffa137ffedc1b131fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "777a8c1010a34e578bcb7861ab6cd9f8" member_type: VOTER last_known_addr { host: "127.3.131.129" port: 39835 } } peers { permanent_uuid: "bdef0b77b17f45ffa137ffedc1b131fe" member_type: VOTER last_known_addr { host: "127.3.131.130" port: 45893 } } peers { permanent_uuid: "739f2ec1059e42e0824f47dbf1912308" member_type: VOTER last_known_addr { host: "127.3.131.131" port: 34855 } }
I20260621 05:14:25.694229 23754 raft_consensus.cc:374] T 6506986b8cee49ecac503df598aec03c P bdef0b77b17f45ffa137ffedc1b131fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 05:14:25.694290 23754 raft_consensus.cc:729] T 6506986b8cee49ecac503df598aec03c P bdef0b77b17f45ffa137ffedc1b131fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bdef0b77b17f45ffa137ffedc1b131fe, State: Initialized, Role: FOLLOWER
I20260621 05:14:25.694429 23754 consensus_queue.cc:260] T 6506986b8cee49ecac503df598aec03c P bdef0b77b17f45ffa137ffedc1b131fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "777a8c1010a34e578bcb7861ab6cd9f8" member_type: VOTER last_known_addr { host: "127.3.131.129" port: 39835 } } peers { permanent_uuid: "bdef0b77b17f45ffa137ffedc1b131fe" member_type: VOTER last_known_addr { host: "127.3.131.130" port: 45893 } } peers { permanent_uuid: "739f2ec1059e42e0824f47dbf1912308" member_type: VOTER last_known_addr { host: "127.3.131.131" port: 34855 } }
I20260621 05:14:25.694628 23758 tablet_bootstrap.cc:492] T 6506986b8cee49ecac503df598aec03c P 777a8c1010a34e578bcb7861ab6cd9f8: No bootstrap required, opened a new log
I20260621 05:14:25.694710 23758 ts_tablet_manager.cc:1397] T 6506986b8cee49ecac503df598aec03c P 777a8c1010a34e578bcb7861ab6cd9f8: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260621 05:14:25.694715 23754 ts_tablet_manager.cc:1428] T 6506986b8cee49ecac503df598aec03c P bdef0b77b17f45ffa137ffedc1b131fe: Time spent starting tablet: real 0.004s user 0.000s sys 0.004s
I20260621 05:14:25.694867 23609 heartbeater.cc:503] Master 127.3.131.190:45663 was elected leader, sending a full tablet report...
I20260621 05:14:25.696866 23758 raft_consensus.cc:348] T 6506986b8cee49ecac503df598aec03c P 777a8c1010a34e578bcb7861ab6cd9f8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "777a8c1010a34e578bcb7861ab6cd9f8" member_type: VOTER last_known_addr { host: "127.3.131.129" port: 39835 } } peers { permanent_uuid: "bdef0b77b17f45ffa137ffedc1b131fe" member_type: VOTER last_known_addr { host: "127.3.131.130" port: 45893 } } peers { permanent_uuid: "739f2ec1059e42e0824f47dbf1912308" member_type: VOTER last_known_addr { host: "127.3.131.131" port: 34855 } }
I20260621 05:14:25.697036 23758 raft_consensus.cc:374] T 6506986b8cee49ecac503df598aec03c P 777a8c1010a34e578bcb7861ab6cd9f8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 05:14:25.697085 23758 raft_consensus.cc:729] T 6506986b8cee49ecac503df598aec03c P 777a8c1010a34e578bcb7861ab6cd9f8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 777a8c1010a34e578bcb7861ab6cd9f8, State: Initialized, Role: FOLLOWER
I20260621 05:14:25.697211 23758 consensus_queue.cc:260] T 6506986b8cee49ecac503df598aec03c P 777a8c1010a34e578bcb7861ab6cd9f8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "777a8c1010a34e578bcb7861ab6cd9f8" member_type: VOTER last_known_addr { host: "127.3.131.129" port: 39835 } } peers { permanent_uuid: "bdef0b77b17f45ffa137ffedc1b131fe" member_type: VOTER last_known_addr { host: "127.3.131.130" port: 45893 } } peers { permanent_uuid: "739f2ec1059e42e0824f47dbf1912308" member_type: VOTER last_known_addr { host: "127.3.131.131" port: 34855 } }
I20260621 05:14:25.697481 23758 ts_tablet_manager.cc:1428] T 6506986b8cee49ecac503df598aec03c P 777a8c1010a34e578bcb7861ab6cd9f8: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260621 05:14:25.697553 23478 heartbeater.cc:503] Master 127.3.131.190:45663 was elected leader, sending a full tablet report...
I20260621 05:14:25.730893 23762 raft_consensus.cc:482] T 6506986b8cee49ecac503df598aec03c P 777a8c1010a34e578bcb7861ab6cd9f8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260621 05:14:25.731046 23762 raft_consensus.cc:504] T 6506986b8cee49ecac503df598aec03c P 777a8c1010a34e578bcb7861ab6cd9f8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "777a8c1010a34e578bcb7861ab6cd9f8" member_type: VOTER last_known_addr { host: "127.3.131.129" port: 39835 } } peers { permanent_uuid: "bdef0b77b17f45ffa137ffedc1b131fe" member_type: VOTER last_known_addr { host: "127.3.131.130" port: 45893 } } peers { permanent_uuid: "739f2ec1059e42e0824f47dbf1912308" member_type: VOTER last_known_addr { host: "127.3.131.131" port: 34855 } }
I20260621 05:14:25.731568 23762 leader_election.cc:288] T 6506986b8cee49ecac503df598aec03c P 777a8c1010a34e578bcb7861ab6cd9f8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bdef0b77b17f45ffa137ffedc1b131fe (127.3.131.130:45893), 739f2ec1059e42e0824f47dbf1912308 (127.3.131.131:34855)
I20260621 05:14:25.734570 23480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:14:25.731763 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.131.131:34855 (local address 127.3.131.129:52555)
0621 05:14:25.731858 (+ 95us) negotiation.cc:107] Waiting for socket to connect
0621 05:14:25.731872 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0621 05:14:25.731926 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:14:25.732106 (+ 180us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:14:25.732107 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:14:25.732109 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 05:14:25.732184 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:14:25.732187 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:14:25.732897 (+ 710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:14:25.732898 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:14:25.733683 (+ 785us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:14:25.733685 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:14:25.734271 (+ 586us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:14:25.734272 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:14:25.734346 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:14:25.734350 (+ 4us) client_negotiation.cc:769] Sending connection context
0621 05:14:25.734439 (+ 89us) client_negotiation.cc:240] Negotiation successful
0621 05:14:25.734489 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":70,"mutex_wait_us":1}
I20260621 05:14:25.734570 23753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:14:25.731881 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.131.129:52555 (local address 127.3.131.131:34855)
0621 05:14:25.731930 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0621 05:14:25.731932 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:14:25.731968 (+ 36us) server_negotiation.cc:407] Connection header received
0621 05:14:25.732006 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:14:25.732007 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:14:25.732017 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:14:25.732098 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 05:14:25.732271 (+ 173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:14:25.732795 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:14:25.733794 (+ 999us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:14:25.734164 (+ 370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:14:25.734271 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:14:25.734280 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0621 05:14:25.734448 (+ 168us) server_negotiation.cc:299] Negotiation successful
0621 05:14:25.734490 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260621 05:14:25.734730 23752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:14:25.731950 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.131.129:53001 (local address 127.3.131.130:45893)
0621 05:14:25.731991 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0621 05:14:25.731993 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:14:25.732216 (+ 223us) server_negotiation.cc:407] Connection header received
0621 05:14:25.732281 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:14:25.732282 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:14:25.732303 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:14:25.732368 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 05:14:25.732536 (+ 168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:14:25.733031 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:14:25.733977 (+ 946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:14:25.734341 (+ 364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:14:25.734429 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:14:25.734436 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0621 05:14:25.734591 (+ 155us) server_negotiation.cc:299] Negotiation successful
0621 05:14:25.734635 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260621 05:14:25.734943 23694 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6506986b8cee49ecac503df598aec03c" candidate_uuid: "777a8c1010a34e578bcb7861ab6cd9f8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "739f2ec1059e42e0824f47dbf1912308" is_pre_election: true
I20260621 05:14:25.735023 23763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:14:25.731879 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.131.130:45893 (local address 127.3.131.129:53001)
0621 05:14:25.732131 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0621 05:14:25.732142 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0621 05:14:25.732218 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:14:25.732372 (+ 154us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:14:25.732373 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:14:25.732374 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 05:14:25.732457 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:14:25.732459 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:14:25.733125 (+ 666us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:14:25.733126 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:14:25.733880 (+ 754us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:14:25.733882 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:14:25.734430 (+ 548us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:14:25.734431 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:14:25.734507 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:14:25.734510 (+ 3us) client_negotiation.cc:769] Sending connection context
0621 05:14:25.734579 (+ 69us) client_negotiation.cc:240] Negotiation successful
0621 05:14:25.734621 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":236,"thread_start_us":161,"threads_started":1}
I20260621 05:14:25.735280 23694 raft_consensus.cc:2454] T 6506986b8cee49ecac503df598aec03c P 739f2ec1059e42e0824f47dbf1912308 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 777a8c1010a34e578bcb7861ab6cd9f8 in term 0.
I20260621 05:14:25.735399 23563 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6506986b8cee49ecac503df598aec03c" candidate_uuid: "777a8c1010a34e578bcb7861ab6cd9f8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bdef0b77b17f45ffa137ffedc1b131fe" is_pre_election: true
I20260621 05:14:25.735561 23563 raft_consensus.cc:2454] T 6506986b8cee49ecac503df598aec03c P bdef0b77b17f45ffa137ffedc1b131fe [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 777a8c1010a34e578bcb7861ab6cd9f8 in term 0.
I20260621 05:14:25.735637 23365 leader_election.cc:302] T 6506986b8cee49ecac503df598aec03c P 777a8c1010a34e578bcb7861ab6cd9f8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 739f2ec1059e42e0824f47dbf1912308, 777a8c1010a34e578bcb7861ab6cd9f8; no voters:
I20260621 05:14:25.735781 23762 raft_consensus.cc:2781] T 6506986b8cee49ecac503df598aec03c P 777a8c1010a34e578bcb7861ab6cd9f8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260621 05:14:25.735878 23762 raft_consensus.cc:482] T 6506986b8cee49ecac503df598aec03c P 777a8c1010a34e578bcb7861ab6cd9f8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260621 05:14:25.735900 23762 raft_consensus.cc:3037] T 6506986b8cee49ecac503df598aec03c P 777a8c1010a34e578bcb7861ab6cd9f8 [term 0 FOLLOWER]: Advancing to term 1
I20260621 05:14:25.736676 23762 raft_consensus.cc:504] T 6506986b8cee49ecac503df598aec03c P 777a8c1010a34e578bcb7861ab6cd9f8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "777a8c1010a34e578bcb7861ab6cd9f8" member_type: VOTER last_known_addr { host: "127.3.131.129" port: 39835 } } peers { permanent_uuid: "bdef0b77b17f45ffa137ffedc1b131fe" member_type: VOTER last_known_addr { host: "127.3.131.130" port: 45893 } } peers { permanent_uuid: "739f2ec1059e42e0824f47dbf1912308" member_type: VOTER last_known_addr { host: "127.3.131.131" port: 34855 } }
I20260621 05:14:25.736917 23762 leader_election.cc:288] T 6506986b8cee49ecac503df598aec03c P 777a8c1010a34e578bcb7861ab6cd9f8 [CANDIDATE]: Term 1 election: Requested vote from peers bdef0b77b17f45ffa137ffedc1b131fe (127.3.131.130:45893), 739f2ec1059e42e0824f47dbf1912308 (127.3.131.131:34855)
I20260621 05:14:25.737002 23563 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6506986b8cee49ecac503df598aec03c" candidate_uuid: "777a8c1010a34e578bcb7861ab6cd9f8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bdef0b77b17f45ffa137ffedc1b131fe"
I20260621 05:14:25.737051 23694 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6506986b8cee49ecac503df598aec03c" candidate_uuid: "777a8c1010a34e578bcb7861ab6cd9f8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "739f2ec1059e42e0824f47dbf1912308"
I20260621 05:14:25.737133 23694 raft_consensus.cc:3037] T 6506986b8cee49ecac503df598aec03c P 739f2ec1059e42e0824f47dbf1912308 [term 0 FOLLOWER]: Advancing to term 1
I20260621 05:14:25.737102 23563 raft_consensus.cc:3037] T 6506986b8cee49ecac503df598aec03c P bdef0b77b17f45ffa137ffedc1b131fe [term 0 FOLLOWER]: Advancing to term 1
I20260621 05:14:25.738176 23694 raft_consensus.cc:2454] T 6506986b8cee49ecac503df598aec03c P 739f2ec1059e42e0824f47dbf1912308 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 777a8c1010a34e578bcb7861ab6cd9f8 in term 1.
I20260621 05:14:25.738176 23563 raft_consensus.cc:2454] T 6506986b8cee49ecac503df598aec03c P bdef0b77b17f45ffa137ffedc1b131fe [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 777a8c1010a34e578bcb7861ab6cd9f8 in term 1.
I20260621 05:14:25.738394 23368 leader_election.cc:302] T 6506986b8cee49ecac503df598aec03c P 777a8c1010a34e578bcb7861ab6cd9f8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 777a8c1010a34e578bcb7861ab6cd9f8, bdef0b77b17f45ffa137ffedc1b131fe; no voters:
I20260621 05:14:25.738574 23762 raft_consensus.cc:2781] T 6506986b8cee49ecac503df598aec03c P 777a8c1010a34e578bcb7861ab6cd9f8 [term 1 FOLLOWER]: Leader election won for term 1
I20260621 05:14:25.738767 23762 raft_consensus.cc:686] T 6506986b8cee49ecac503df598aec03c P 777a8c1010a34e578bcb7861ab6cd9f8 [term 1 LEADER]: Becoming Leader. State: Replica: 777a8c1010a34e578bcb7861ab6cd9f8, State: Running, Role: LEADER
I20260621 05:14:25.738886 23762 consensus_queue.cc:237] T 6506986b8cee49ecac503df598aec03c P 777a8c1010a34e578bcb7861ab6cd9f8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "777a8c1010a34e578bcb7861ab6cd9f8" member_type: VOTER last_known_addr { host: "127.3.131.129" port: 39835 } } peers { permanent_uuid: "bdef0b77b17f45ffa137ffedc1b131fe" member_type: VOTER last_known_addr { host: "127.3.131.130" port: 45893 } } peers { permanent_uuid: "739f2ec1059e42e0824f47dbf1912308" member_type: VOTER last_known_addr { host: "127.3.131.131" port: 34855 } }
I20260621 05:14:25.739888 23290 catalog_manager.cc:5495] T 6506986b8cee49ecac503df598aec03c P 777a8c1010a34e578bcb7861ab6cd9f8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 777a8c1010a34e578bcb7861ab6cd9f8 (127.3.131.129). New cstate: current_term: 1 leader_uuid: "777a8c1010a34e578bcb7861ab6cd9f8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "777a8c1010a34e578bcb7861ab6cd9f8" member_type: VOTER last_known_addr { host: "127.3.131.129" port: 39835 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bdef0b77b17f45ffa137ffedc1b131fe" member_type: VOTER last_known_addr { host: "127.3.131.130" port: 45893 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "739f2ec1059e42e0824f47dbf1912308" member_type: VOTER last_known_addr { host: "127.3.131.131" port: 34855 } health_report { overall_health: UNKNOWN } } }
W20260621 05:14:25.759266 23610 tablet.cc:2369] T 6506986b8cee49ecac503df598aec03c P bdef0b77b17f45ffa137ffedc1b131fe: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260621 05:14:25.759799 23751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:14:25.757167 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43140 (local address 127.3.131.129:39835)
0621 05:14:25.757218 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0621 05:14:25.757220 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:14:25.757226 (+ 6us) server_negotiation.cc:407] Connection header received
0621 05:14:25.757263 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:14:25.757264 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:14:25.757282 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:14:25.757337 (+ 55us) server_negotiation.cc:226] Negotiated authn=TOKEN
0621 05:14:25.757495 (+ 158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:14:25.758040 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:14:25.758862 (+ 822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:14:25.759106 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:14:25.759204 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:14:25.759423 (+ 219us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0621 05:14:25.759557 (+ 134us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0621 05:14:25.759635 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0621 05:14:25.759672 (+ 37us) server_negotiation.cc:299] Negotiation successful
0621 05:14:25.759690 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260621 05:14:25.761691 23563 raft_consensus.cc:1261] T 6506986b8cee49ecac503df598aec03c P bdef0b77b17f45ffa137ffedc1b131fe [term 1 FOLLOWER]: Refusing update from remote peer 777a8c1010a34e578bcb7861ab6cd9f8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 05:14:25.761698 23694 raft_consensus.cc:1261] T 6506986b8cee49ecac503df598aec03c P 739f2ec1059e42e0824f47dbf1912308 [term 1 FOLLOWER]: Refusing update from remote peer 777a8c1010a34e578bcb7861ab6cd9f8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 05:14:25.762137 23764 consensus_queue.cc:1035] T 6506986b8cee49ecac503df598aec03c P 777a8c1010a34e578bcb7861ab6cd9f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "739f2ec1059e42e0824f47dbf1912308" member_type: VOTER last_known_addr { host: "127.3.131.131" port: 34855 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 05:14:25.762296 23762 consensus_queue.cc:1035] T 6506986b8cee49ecac503df598aec03c P 777a8c1010a34e578bcb7861ab6cd9f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bdef0b77b17f45ffa137ffedc1b131fe" member_type: VOTER last_known_addr { host: "127.3.131.130" port: 45893 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 05:14:25.764756 23767 mvcc.cc:205] Tried to move back new op lower bound from 7299149274156957696 to 7299149274067128320. Current Snapshot: MvccSnapshot[applied={T|T < 7299149274156957696}]
I20260621 05:14:25.765055 23769 mvcc.cc:205] Tried to move back new op lower bound from 7299149274156957696 to 7299149274067128320. Current Snapshot: MvccSnapshot[applied={T|T < 7299149274156957696}]
I20260621 05:14:25.765488 23768 mvcc.cc:205] Tried to move back new op lower bound from 7299149274156957696 to 7299149274067128320. Current Snapshot: MvccSnapshot[applied={T|T < 7299149274156957696}]
I20260621 05:14:25.767445 23290 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:46140:
table { table_name: "test-table" } modify_external_catalogs: true
I20260621 05:14:25.767562 23290 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:46140:
table { table_name: "test-table" } modify_external_catalogs: true
I20260621 05:14:25.768384 23290 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9a269880f1924c1ba2c4619ff627567d: Sending DeleteTablet for 3 replicas of tablet 6506986b8cee49ecac503df598aec03c
I20260621 05:14:25.768741 23674 tablet_service.cc:1514] Processing DeleteTablet for tablet 6506986b8cee49ecac503df598aec03c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 05:14:25 UTC) from {username='slave'} at 127.0.0.1:51408
I20260621 05:14:25.768780 23412 tablet_service.cc:1514] Processing DeleteTablet for tablet 6506986b8cee49ecac503df598aec03c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 05:14:25 UTC) from {username='slave'} at 127.0.0.1:43128
I20260621 05:14:25.768796 23543 tablet_service.cc:1514] Processing DeleteTablet for tablet 6506986b8cee49ecac503df598aec03c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 05:14:25 UTC) from {username='slave'} at 127.0.0.1:37380
I20260621 05:14:25.769109 23778 tablet_replica.cc:330] T 6506986b8cee49ecac503df598aec03c P bdef0b77b17f45ffa137ffedc1b131fe: stopping tablet replica
I20260621 05:14:25.769137 23777 tablet_replica.cc:330] T 6506986b8cee49ecac503df598aec03c P 739f2ec1059e42e0824f47dbf1912308: stopping tablet replica
I20260621 05:14:25.769302 3598 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWZOx2_/build/release/bin/kudu with pid 23350
I20260621 05:14:25.769939 23777 raft_consensus.cc:2229] T 6506986b8cee49ecac503df598aec03c P 739f2ec1059e42e0824f47dbf1912308 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260621 05:14:25.769232 23778 raft_consensus.cc:2229] T 6506986b8cee49ecac503df598aec03c P bdef0b77b17f45ffa137ffedc1b131fe [term 1 FOLLOWER]: Raft consensus shutting down.
I20260621 05:14:25.770196 23777 raft_consensus.cc:2258] T 6506986b8cee49ecac503df598aec03c P 739f2ec1059e42e0824f47dbf1912308 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 05:14:25.770231 23778 raft_consensus.cc:2258] T 6506986b8cee49ecac503df598aec03c P bdef0b77b17f45ffa137ffedc1b131fe [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 05:14:25.770720 23777 ts_tablet_manager.cc:1905] T 6506986b8cee49ecac503df598aec03c P 739f2ec1059e42e0824f47dbf1912308: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 05:14:25.770720 23778 ts_tablet_manager.cc:1905] T 6506986b8cee49ecac503df598aec03c P bdef0b77b17f45ffa137ffedc1b131fe: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 05:14:25.772585 23778 ts_tablet_manager.cc:1918] T 6506986b8cee49ecac503df598aec03c P bdef0b77b17f45ffa137ffedc1b131fe: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260621 05:14:25.772624 23777 ts_tablet_manager.cc:1918] T 6506986b8cee49ecac503df598aec03c P 739f2ec1059e42e0824f47dbf1912308: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260621 05:14:25.772679 23778 log.cc:1196] T 6506986b8cee49ecac503df598aec03c P bdef0b77b17f45ffa137ffedc1b131fe: Deleting WAL directory at /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782018616648578-3598-0/minicluster-data/ts-1/wal/wals/6506986b8cee49ecac503df598aec03c
I20260621 05:14:25.772692 23777 log.cc:1196] T 6506986b8cee49ecac503df598aec03c P 739f2ec1059e42e0824f47dbf1912308: Deleting WAL directory at /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782018616648578-3598-0/minicluster-data/ts-2/wal/wals/6506986b8cee49ecac503df598aec03c
I20260621 05:14:25.772970 23777 ts_tablet_manager.cc:1939] T 6506986b8cee49ecac503df598aec03c P 739f2ec1059e42e0824f47dbf1912308: Deleting consensus metadata
I20260621 05:14:25.772974 23778 ts_tablet_manager.cc:1939] T 6506986b8cee49ecac503df598aec03c P bdef0b77b17f45ffa137ffedc1b131fe: Deleting consensus metadata
I20260621 05:14:25.773516 23277 catalog_manager.cc:4841] TS bdef0b77b17f45ffa137ffedc1b131fe (127.3.131.130:45893): tablet 6506986b8cee49ecac503df598aec03c (table test-table [id=88cefc4f787543e899fa5579200dd020]) successfully deleted
I20260621 05:14:25.773641 23274 catalog_manager.cc:4841] TS 739f2ec1059e42e0824f47dbf1912308 (127.3.131.131:34855): tablet 6506986b8cee49ecac503df598aec03c (table test-table [id=88cefc4f787543e899fa5579200dd020]) successfully deleted
W20260621 05:14:25.775590 23275 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.3.131.129:39835 (error 108)
I20260621 05:14:25.775942 3598 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWZOx2_/build/release/bin/kudu with pid 23481
I20260621 05:14:25.776249 23748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:14:25.776094 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.131.129:39835 (local address 127.0.0.1:43148)
0621 05:14:25.776153 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0621 05:14:25.776196 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.131.129:39835: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260621 05:14:25.776363 23275 catalog_manager.cc:4584] TS 777a8c1010a34e578bcb7861ab6cd9f8 (127.3.131.129:39835): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 6506986b8cee49ecac503df598aec03c: Network error: Client connection negotiation failed: client connection to 127.3.131.129:39835: connect: Connection refused (error 111)
I20260621 05:14:25.780803 3598 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWZOx2_/build/release/bin/kudu with pid 23612
I20260621 05:14:25.785830 3598 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWZOx2_/build/release/bin/kudu with pid 23259
2026-06-21T05:14:25Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (760 ms)
[----------] 6 tests from EncryptionPolicyTest (8472 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1782018616648578-3598-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 21 05:14:25 dist-test-slave-vrq1 krb5kdc[23783](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 21 05:14:25 dist-test-slave-vrq1 krb5kdc[23783](info): set up 2 sockets
Jun 21 05:14:25 dist-test-slave-vrq1 krb5kdc[23783](info): commencing operation
krb5kdc: starting...
W20260621 05:14:26.879637 3598 mini_kdc.cc:121] Time spent starting KDC: real 1.074s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1782018616648578-3598-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1782018616648578-3598-0/krb5kdc/test-user.keytab.
Jun 21 05:14:26 dist-test-slave-vrq1 krb5kdc[23783](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782018866, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-21T05:14:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-21T05:14:26Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1782018616648578-3598-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1782018616648578-3598-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260621 05:14:27.020713 3598 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWZOx2_/build/release/bin/kudu
/tmp/dist-test-taskWZOx2_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1782018616648578-3598-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1782018616648578-3598-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1782018616648578-3598-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1782018616648578-3598-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35269
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1782018616648578-3598-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:41883
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:35269
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1782018616648578-3598-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1782018616648578-3598-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1782018616648578-3598-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 05:14:27.102977 23796 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 05:14:27.103260 23796 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 05:14:27.103286 23796 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 05:14:27.104430 23796 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 05:14:27.104467 23796 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 05:14:27.104482 23796 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 05:14:27.104496 23796 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 05:14:27.104507 23796 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 05:14:27.105947 23796 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:41883
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1782018616648578-3598-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1782018616648578-3598-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:35269
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1782018616648578-3598-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35269
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1782018616648578-3598-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23796
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1782018616648578-3598-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 21 Jun 2026 04:16:16 UTC on ccycloud.quasar-xycaha.root.comops.site
build id 4457881
I20260621 05:14:27.106159 23796 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 05:14:27.106336 23796 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 05:14:27.111567 23804 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:14:27.111569 23801 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:14:27.111582 23802 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:14:27.112073 23796 server_base.cc:1034] running on GCE node
I20260621 05:14:27.112346 23796 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 05:14:27.112720 23796 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 05:14:27.113889 23796 hybrid_clock.cc:648] HybridClock initialized: now 1782018867113852 us; error 58 us; skew 500 ppm
Jun 21 05:14:27 dist-test-slave-vrq1 krb5kdc[23783](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782018867, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 05:14:27.121196 23796 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260621 05:14:27.121747 23796 webserver.cc:460] Webserver started at http://127.0.0.1:34585/ using document root <none> and password file <none>
I20260621 05:14:27.122013 23796 fs_manager.cc:359] Metadata directory not provided
I20260621 05:14:27.122061 23796 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 05:14:27.122241 23796 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 05:14:27.123432 23796 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1782018616648578-3598-0/minicluster-data/master-0/data/instance:
uuid: "968790e81be143b5ae017cdfb0b6449c"
format_stamp: "Formatted at 2026-06-21 05:14:27 on dist-test-slave-vrq1"
server_key: "95900c3744880d4ee71da97845cda182"
server_key_iv: "0185ac3165a0ddedfd79bf0bd09ce6bb"
server_key_version: "encryptionkey@0"
I20260621 05:14:27.124053 23796 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1782018616648578-3598-0/minicluster-data/master-0/wal/instance:
uuid: "968790e81be143b5ae017cdfb0b6449c"
format_stamp: "Formatted at 2026-06-21 05:14:27 on dist-test-slave-vrq1"
server_key: "95900c3744880d4ee71da97845cda182"
server_key_iv: "0185ac3165a0ddedfd79bf0bd09ce6bb"
server_key_version: "encryptionkey@0"
I20260621 05:14:27.125649 23796 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260621 05:14:27.126484 23811 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 05:14:27.126821 23796 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260621 05:14:27.126907 23796 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1782018616648578-3598-0/minicluster-data/master-0/data,/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1782018616648578-3598-0/minicluster-data/master-0/wal
uuid: "968790e81be143b5ae017cdfb0b6449c"
format_stamp: "Formatted at 2026-06-21 05:14:27 on dist-test-slave-vrq1"
server_key: "95900c3744880d4ee71da97845cda182"
server_key_iv: "0185ac3165a0ddedfd79bf0bd09ce6bb"
server_key_version: "encryptionkey@0"
I20260621 05:14:27.127027 23796 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1782018616648578-3598-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1782018616648578-3598-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1782018616648578-3598-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 05:14:27.144398 23796 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 05:14:27.146675 23796 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 05:14:27.146831 23796 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 05:14:27.151564 23796 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:35269
I20260621 05:14:27.151669 23863 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:35269 every 8 connection(s)
I20260621 05:14:27.152040 23796 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1782018616648578-3598-0/minicluster-data/master-0/data/info.pb
I20260621 05:14:27.153107 23864 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 05:14:27.155791 23864 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 968790e81be143b5ae017cdfb0b6449c: Bootstrap starting.
I20260621 05:14:27.156430 23864 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 968790e81be143b5ae017cdfb0b6449c: Neither blocks nor log segments found. Creating new log.
I20260621 05:14:27.156755 3598 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWZOx2_/build/release/bin/kudu as pid 23796
I20260621 05:14:27.156774 23864 log.cc:824] T 00000000000000000000000000000000 P 968790e81be143b5ae017cdfb0b6449c: Log is configured to *not* fsync() on all Append() calls
I20260621 05:14:27.156898 3598 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1782018616648578-3598-0/minicluster-data/master-0/wal/instance
I20260621 05:14:27.157119 3598 external_mini_cluster.cc:1428] Setting key bfba261d6ea22764cd3783526fe78ba8
I20260621 05:14:27.157850 23864 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 968790e81be143b5ae017cdfb0b6449c: No bootstrap required, opened a new log
I20260621 05:14:27.160006 23864 raft_consensus.cc:348] T 00000000000000000000000000000000 P 968790e81be143b5ae017cdfb0b6449c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "968790e81be143b5ae017cdfb0b6449c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35269 } }
I20260621 05:14:27.160202 23864 raft_consensus.cc:374] T 00000000000000000000000000000000 P 968790e81be143b5ae017cdfb0b6449c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 05:14:27.160238 23864 raft_consensus.cc:729] T 00000000000000000000000000000000 P 968790e81be143b5ae017cdfb0b6449c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 968790e81be143b5ae017cdfb0b6449c, State: Initialized, Role: FOLLOWER
I20260621 05:14:27.160400 23864 consensus_queue.cc:260] T 00000000000000000000000000000000 P 968790e81be143b5ae017cdfb0b6449c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "968790e81be143b5ae017cdfb0b6449c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35269 } }
I20260621 05:14:27.160506 23864 raft_consensus.cc:388] T 00000000000000000000000000000000 P 968790e81be143b5ae017cdfb0b6449c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260621 05:14:27.160557 23864 raft_consensus.cc:482] T 00000000000000000000000000000000 P 968790e81be143b5ae017cdfb0b6449c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260621 05:14:27.160611 23864 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 968790e81be143b5ae017cdfb0b6449c [term 0 FOLLOWER]: Advancing to term 1
I20260621 05:14:27.161375 23864 raft_consensus.cc:504] T 00000000000000000000000000000000 P 968790e81be143b5ae017cdfb0b6449c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "968790e81be143b5ae017cdfb0b6449c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35269 } }
I20260621 05:14:27.161490 23864 leader_election.cc:302] T 00000000000000000000000000000000 P 968790e81be143b5ae017cdfb0b6449c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 968790e81be143b5ae017cdfb0b6449c; no voters:
I20260621 05:14:27.161664 23864 leader_election.cc:288] T 00000000000000000000000000000000 P 968790e81be143b5ae017cdfb0b6449c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260621 05:14:27.161742 23869 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 968790e81be143b5ae017cdfb0b6449c [term 1 FOLLOWER]: Leader election won for term 1
I20260621 05:14:27.161922 23864 sys_catalog.cc:564] T 00000000000000000000000000000000 P 968790e81be143b5ae017cdfb0b6449c [sys.catalog]: configured and running, proceeding with master startup.
I20260621 05:14:27.161922 23869 raft_consensus.cc:686] T 00000000000000000000000000000000 P 968790e81be143b5ae017cdfb0b6449c [term 1 LEADER]: Becoming Leader. State: Replica: 968790e81be143b5ae017cdfb0b6449c, State: Running, Role: LEADER
I20260621 05:14:27.162123 23869 consensus_queue.cc:237] T 00000000000000000000000000000000 P 968790e81be143b5ae017cdfb0b6449c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "968790e81be143b5ae017cdfb0b6449c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35269 } }
I20260621 05:14:27.162528 23870 sys_catalog.cc:455] T 00000000000000000000000000000000 P 968790e81be143b5ae017cdfb0b6449c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "968790e81be143b5ae017cdfb0b6449c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "968790e81be143b5ae017cdfb0b6449c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35269 } } }
I20260621 05:14:27.162633 23870 sys_catalog.cc:458] T 00000000000000000000000000000000 P 968790e81be143b5ae017cdfb0b6449c [sys.catalog]: This master's current role is: LEADER
I20260621 05:14:27.163332 23871 sys_catalog.cc:455] T 00000000000000000000000000000000 P 968790e81be143b5ae017cdfb0b6449c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 968790e81be143b5ae017cdfb0b6449c. Latest consensus state: current_term: 1 leader_uuid: "968790e81be143b5ae017cdfb0b6449c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "968790e81be143b5ae017cdfb0b6449c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35269 } } }
I20260621 05:14:27.163595 23871 sys_catalog.cc:458] T 00000000000000000000000000000000 P 968790e81be143b5ae017cdfb0b6449c [sys.catalog]: This master's current role is: LEADER
W20260621 05:14:27.164072 23884 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 968790e81be143b5ae017cdfb0b6449c: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260621 05:14:27.164135 23884 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260621 05:14:27.164197 23878 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260621 05:14:27.164402 23878 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Jun 21 05:14:27 dist-test-slave-vrq1 krb5kdc[23783](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782018866, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260621 05:14:27.170919 23878 catalog_manager.cc:1347] Generated new cluster ID: 37486516b26b46539365d60db716d0c7
I20260621 05:14:27.170986 23878 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260621 05:14:27.172011 23868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:14:27.157718 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32961 (local address 127.0.0.1:35269)
0621 05:14:27.157920 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0621 05:14:27.157930 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0621 05:14:27.157970 (+ 40us) server_negotiation.cc:407] Connection header received
0621 05:14:27.159187 (+ 1217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:14:27.159211 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:14:27.159283 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:14:27.159452 (+ 169us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:14:27.160182 (+ 730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:14:27.161929 (+ 1747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:14:27.163093 (+ 1164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:14:27.163764 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:14:27.163890 (+ 126us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:14:27.167746 (+ 3856us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:14:27.167753 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:14:27.167753 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:14:27.167767 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:14:27.169894 (+ 2127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:14:27.170392 (+ 498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:14:27.170396 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:14:27.170396 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:14:27.170521 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:14:27.170842 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:14:27.170843 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:14:27.170844 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:14:27.171193 (+ 349us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:14:27.171361 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0621 05:14:27.171523 (+ 162us) server_negotiation.cc:299] Negotiation successful
0621 05:14:27.171796 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":144,"thread_start_us":86,"threads_started":1}
I20260621 05:14:27.173003 3598 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260621 05:14:27.182222 23868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:14:27.174542 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58449 (local address 127.0.0.1:35269)
0621 05:14:27.174585 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0621 05:14:27.174586 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0621 05:14:27.174722 (+ 136us) server_negotiation.cc:407] Connection header received
0621 05:14:27.174786 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:14:27.174787 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:14:27.174797 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:14:27.174853 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:14:27.175604 (+ 751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:14:27.176365 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:14:27.177424 (+ 1059us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:14:27.177737 (+ 313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:14:27.177795 (+ 58us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:14:27.179354 (+ 1559us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:14:27.179356 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:14:27.179356 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:14:27.179363 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:14:27.180954 (+ 1591us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:14:27.181400 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:14:27.181401 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:14:27.181402 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:14:27.181458 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:14:27.181689 (+ 231us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:14:27.181689 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:14:27.181690 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:14:27.181810 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:14:27.181899 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0621 05:14:27.182064 (+ 165us) server_negotiation.cc:299] Negotiation successful
0621 05:14:27.182127 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260621 05:14:27.188109 23878 catalog_manager.cc:1370] Generated new certificate authority record
I20260621 05:14:27.188886 23878 catalog_manager.cc:1504] Loading token signing keys...
I20260621 05:14:27.196297 23878 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 968790e81be143b5ae017cdfb0b6449c: Generated new TSK 0
I20260621 05:14:27.196543 23878 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260621 05:14:27.201913 3598 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWZOx2_/build/release/bin/kudu with pid 23796
W20260621 05:14:27.207453 23781 connection.cc:535] client connection to 127.0.0.1:35269 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:35269): Connection reset by peer (error 104)
2026-06-21T05:14:27Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (1415 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1782018616648578-3598-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 21 05:14:27 dist-test-slave-vrq1 krb5kdc[23895](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 21 05:14:27 dist-test-slave-vrq1 krb5kdc[23895](info): set up 2 sockets
Jun 21 05:14:27 dist-test-slave-vrq1 krb5kdc[23895](info): commencing operation
krb5kdc: starting...
W20260621 05:14:28.323822 3598 mini_kdc.cc:121] Time spent starting KDC: real 1.102s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1782018616648578-3598-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1782018616648578-3598-0/krb5kdc/test-user.keytab.
Jun 21 05:14:28 dist-test-slave-vrq1 krb5kdc[23895](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782018868, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-21T05:14:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-21T05:14:28Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1782018616648578-3598-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1782018616648578-3598-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260621 05:14:28.464982 3598 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWZOx2_/build/release/bin/kudu
/tmp/dist-test-taskWZOx2_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1782018616648578-3598-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1782018616648578-3598-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1782018616648578-3598-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1782018616648578-3598-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39145
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1782018616648578-3598-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:37119
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:39145
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1782018616648578-3598-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1782018616648578-3598-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1782018616648578-3598-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 05:14:28.558996 23908 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 05:14:28.559332 23908 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 05:14:28.559379 23908 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 05:14:28.560690 23908 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 05:14:28.560772 23908 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 05:14:28.560796 23908 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 05:14:28.560817 23908 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 05:14:28.560847 23908 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 05:14:28.562539 23908 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:37119
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1782018616648578-3598-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1782018616648578-3598-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:39145
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1782018616648578-3598-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39145
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1782018616648578-3598-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23908
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1782018616648578-3598-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 21 Jun 2026 04:16:16 UTC on ccycloud.quasar-xycaha.root.comops.site
build id 4457881
I20260621 05:14:28.562857 23908 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 05:14:28.563108 23908 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 05:14:28.568806 23916 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:14:28.568843 23913 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:14:28.568806 23914 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:14:28.569033 23908 server_base.cc:1034] running on GCE node
I20260621 05:14:28.569423 23908 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 05:14:28.569717 23908 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 05:14:28.570971 23908 hybrid_clock.cc:648] HybridClock initialized: now 1782018868570916 us; error 76 us; skew 500 ppm
Jun 21 05:14:28 dist-test-slave-vrq1 krb5kdc[23895](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782018868, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 05:14:28.574036 23908 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260621 05:14:28.574481 23908 webserver.cc:460] Webserver started at http://127.0.0.1:44293/ using document root <none> and password file <none>
I20260621 05:14:28.574743 23908 fs_manager.cc:359] Metadata directory not provided
I20260621 05:14:28.574795 23908 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 05:14:28.574926 23908 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 05:14:28.576102 23908 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1782018616648578-3598-0/minicluster-data/master-0/data/instance:
uuid: "5d6c06befaad4fc4a9c29c0366947ead"
format_stamp: "Formatted at 2026-06-21 05:14:28 on dist-test-slave-vrq1"
server_key: "f4753cfadccbe0d40d0ae452abcb615b"
server_key_iv: "c83402d0fe89b760d22116a026f60a87"
server_key_version: "encryptionkey@0"
I20260621 05:14:28.576488 23908 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1782018616648578-3598-0/minicluster-data/master-0/wal/instance:
uuid: "5d6c06befaad4fc4a9c29c0366947ead"
format_stamp: "Formatted at 2026-06-21 05:14:28 on dist-test-slave-vrq1"
server_key: "f4753cfadccbe0d40d0ae452abcb615b"
server_key_iv: "c83402d0fe89b760d22116a026f60a87"
server_key_version: "encryptionkey@0"
I20260621 05:14:28.578349 23908 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.002s sys 0.000s
I20260621 05:14:28.579492 23923 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 05:14:28.579782 23908 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260621 05:14:28.580003 23908 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1782018616648578-3598-0/minicluster-data/master-0/data,/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1782018616648578-3598-0/minicluster-data/master-0/wal
uuid: "5d6c06befaad4fc4a9c29c0366947ead"
format_stamp: "Formatted at 2026-06-21 05:14:28 on dist-test-slave-vrq1"
server_key: "f4753cfadccbe0d40d0ae452abcb615b"
server_key_iv: "c83402d0fe89b760d22116a026f60a87"
server_key_version: "encryptionkey@0"
I20260621 05:14:28.580096 23908 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1782018616648578-3598-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1782018616648578-3598-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1782018616648578-3598-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 05:14:28.596202 23908 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 05:14:28.598999 23908 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 05:14:28.599277 23908 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 05:14:28.603502 23908 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:39145
I20260621 05:14:28.603534 23975 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:39145 every 8 connection(s)
I20260621 05:14:28.604193 23908 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1782018616648578-3598-0/minicluster-data/master-0/data/info.pb
I20260621 05:14:28.605535 23976 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 05:14:28.608078 23976 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5d6c06befaad4fc4a9c29c0366947ead: Bootstrap starting.
I20260621 05:14:28.608637 23976 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5d6c06befaad4fc4a9c29c0366947ead: Neither blocks nor log segments found. Creating new log.
I20260621 05:14:28.608911 23976 log.cc:824] T 00000000000000000000000000000000 P 5d6c06befaad4fc4a9c29c0366947ead: Log is configured to *not* fsync() on all Append() calls
I20260621 05:14:28.609711 23976 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5d6c06befaad4fc4a9c29c0366947ead: No bootstrap required, opened a new log
I20260621 05:14:28.611364 3598 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWZOx2_/build/release/bin/kudu as pid 23908
I20260621 05:14:28.611508 3598 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1782018616648578-3598-0/minicluster-data/master-0/wal/instance
I20260621 05:14:28.611744 3598 external_mini_cluster.cc:1428] Setting key de5f16d0f6e1cafe2720ce7881e14b71
I20260621 05:14:28.612236 23976 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5d6c06befaad4fc4a9c29c0366947ead [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5d6c06befaad4fc4a9c29c0366947ead" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39145 } }
I20260621 05:14:28.612533 23976 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5d6c06befaad4fc4a9c29c0366947ead [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 05:14:28.612627 23976 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5d6c06befaad4fc4a9c29c0366947ead [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5d6c06befaad4fc4a9c29c0366947ead, State: Initialized, Role: FOLLOWER
I20260621 05:14:28.612813 23976 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5d6c06befaad4fc4a9c29c0366947ead [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5d6c06befaad4fc4a9c29c0366947ead" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39145 } }
I20260621 05:14:28.612922 23976 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5d6c06befaad4fc4a9c29c0366947ead [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260621 05:14:28.613026 23976 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5d6c06befaad4fc4a9c29c0366947ead [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260621 05:14:28.613094 23976 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5d6c06befaad4fc4a9c29c0366947ead [term 0 FOLLOWER]: Advancing to term 1
I20260621 05:14:28.613905 23976 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5d6c06befaad4fc4a9c29c0366947ead [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5d6c06befaad4fc4a9c29c0366947ead" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39145 } }
I20260621 05:14:28.614341 23976 leader_election.cc:302] T 00000000000000000000000000000000 P 5d6c06befaad4fc4a9c29c0366947ead [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5d6c06befaad4fc4a9c29c0366947ead; no voters:
I20260621 05:14:28.614565 23976 leader_election.cc:288] T 00000000000000000000000000000000 P 5d6c06befaad4fc4a9c29c0366947ead [CANDIDATE]: Term 1 election: Requested vote from peers
I20260621 05:14:28.614621 23981 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5d6c06befaad4fc4a9c29c0366947ead [term 1 FOLLOWER]: Leader election won for term 1
I20260621 05:14:28.614748 23981 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5d6c06befaad4fc4a9c29c0366947ead [term 1 LEADER]: Becoming Leader. State: Replica: 5d6c06befaad4fc4a9c29c0366947ead, State: Running, Role: LEADER
I20260621 05:14:28.614868 23976 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5d6c06befaad4fc4a9c29c0366947ead [sys.catalog]: configured and running, proceeding with master startup.
I20260621 05:14:28.614897 23981 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5d6c06befaad4fc4a9c29c0366947ead [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5d6c06befaad4fc4a9c29c0366947ead" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39145 } }
I20260621 05:14:28.615545 23981 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5d6c06befaad4fc4a9c29c0366947ead [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5d6c06befaad4fc4a9c29c0366947ead. Latest consensus state: current_term: 1 leader_uuid: "5d6c06befaad4fc4a9c29c0366947ead" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5d6c06befaad4fc4a9c29c0366947ead" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39145 } } }
I20260621 05:14:28.615680 23981 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5d6c06befaad4fc4a9c29c0366947ead [sys.catalog]: This master's current role is: LEADER
I20260621 05:14:28.616034 23983 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5d6c06befaad4fc4a9c29c0366947ead [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5d6c06befaad4fc4a9c29c0366947ead" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5d6c06befaad4fc4a9c29c0366947ead" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39145 } } }
I20260621 05:14:28.616146 23983 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5d6c06befaad4fc4a9c29c0366947ead [sys.catalog]: This master's current role is: LEADER
I20260621 05:14:28.616318 23988 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260621 05:14:28.617015 23988 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260621 05:14:28.618830 23980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:14:28.612328 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56653 (local address 127.0.0.1:39145)
0621 05:14:28.612541 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0621 05:14:28.612551 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0621 05:14:28.612587 (+ 36us) server_negotiation.cc:407] Connection header received
0621 05:14:28.612718 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:14:28.612725 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:14:28.612778 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:14:28.612893 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:14:28.613275 (+ 382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:14:28.614568 (+ 1293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:14:28.615860 (+ 1292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:14:28.616612 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:14:28.617214 (+ 602us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:14:28.617809 (+ 595us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:14:28.617814 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:14:28.617814 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 05:14:28.618080 (+ 266us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:14:28.618113 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 05:14:28.618125 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:14:28.618234 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0621 05:14:28.618564 (+ 330us) server_negotiation.cc:299] Negotiation successful
0621 05:14:28.618598 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":127,"thread_start_us":96,"threads_started":1}
I20260621 05:14:28.618903 23988 catalog_manager.cc:1347] Generated new cluster ID: ca8e995653d54394ac9ce6047d35e969
I20260621 05:14:28.619135 23988 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260621 05:14:28.620103 3598 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260621 05:14:28.625425 23980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:14:28.621534 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59331 (local address 127.0.0.1:39145)
0621 05:14:28.621574 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0621 05:14:28.621576 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:14:28.621739 (+ 163us) server_negotiation.cc:407] Connection header received
0621 05:14:28.621826 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:14:28.621827 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:14:28.621846 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:14:28.621873 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:14:28.622067 (+ 194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:14:28.622847 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:14:28.623920 (+ 1073us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:14:28.624253 (+ 333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:14:28.624293 (+ 40us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:14:28.625158 (+ 865us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:14:28.625159 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:14:28.625160 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 05:14:28.625165 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:14:28.625179 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 05:14:28.625184 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:14:28.625212 (+ 28us) server_negotiation.cc:1035] Waiting for connection context
0621 05:14:28.625320 (+ 108us) server_negotiation.cc:299] Negotiation successful
0621 05:14:28.625335 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260621 05:14:28.638796 23988 catalog_manager.cc:1370] Generated new certificate authority record
I20260621 05:14:28.639541 23988 catalog_manager.cc:1504] Loading token signing keys...
I20260621 05:14:28.647202 23988 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5d6c06befaad4fc4a9c29c0366947ead: Generated new TSK 0
I20260621 05:14:28.647425 23988 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260621 05:14:28.652140 3598 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWZOx2_/build/release/bin/kudu with pid 23908
W20260621 05:14:28.656632 23893 connection.cc:535] client connection to 127.0.0.1:39145 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:39145): Connection reset by peer (error 104)
2026-06-21T05:14:28Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1449 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1782018616648578-3598-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 21 05:14:28 dist-test-slave-vrq1 krb5kdc[24007](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 21 05:14:28 dist-test-slave-vrq1 krb5kdc[24007](info): set up 2 sockets
Jun 21 05:14:28 dist-test-slave-vrq1 krb5kdc[24007](info): commencing operation
krb5kdc: starting...
W20260621 05:14:29.719599 3598 mini_kdc.cc:121] Time spent starting KDC: real 1.049s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1782018616648578-3598-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1782018616648578-3598-0/krb5kdc/test-user.keytab.
Jun 21 05:14:29 dist-test-slave-vrq1 krb5kdc[24007](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782018869, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-21T05:14:29Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-21T05:14:29Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1782018616648578-3598-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1782018616648578-3598-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260621 05:14:29.850977 3598 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWZOx2_/build/release/bin/kudu
/tmp/dist-test-taskWZOx2_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1782018616648578-3598-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1782018616648578-3598-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1782018616648578-3598-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1782018616648578-3598-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44021
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1782018616648578-3598-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:46021
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:44021
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1782018616648578-3598-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1782018616648578-3598-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1782018616648578-3598-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 05:14:29.940402 24020 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 05:14:29.940651 24020 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 05:14:29.940678 24020 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 05:14:29.942035 24020 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 05:14:29.942099 24020 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 05:14:29.942114 24020 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 05:14:29.942127 24020 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 05:14:29.942140 24020 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 05:14:29.943864 24020 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:46021
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1782018616648578-3598-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1782018616648578-3598-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:44021
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1782018616648578-3598-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44021
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1782018616648578-3598-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24020
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1782018616648578-3598-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 21 Jun 2026 04:16:16 UTC on ccycloud.quasar-xycaha.root.comops.site
build id 4457881
I20260621 05:14:29.944171 24020 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 05:14:29.944446 24020 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 05:14:29.950892 24028 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:14:29.951118 24025 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:14:29.951263 24026 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:14:29.951361 24020 server_base.cc:1034] running on GCE node
I20260621 05:14:29.951887 24020 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 05:14:29.952237 24020 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 05:14:29.953419 24020 hybrid_clock.cc:648] HybridClock initialized: now 1782018869953342 us; error 91 us; skew 500 ppm
Jun 21 05:14:29 dist-test-slave-vrq1 krb5kdc[24007](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782018869, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 05:14:29.956478 24020 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260621 05:14:29.956996 24020 webserver.cc:460] Webserver started at http://127.0.0.1:43391/ using document root <none> and password file <none>
I20260621 05:14:29.957263 24020 fs_manager.cc:359] Metadata directory not provided
I20260621 05:14:29.957337 24020 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 05:14:29.957495 24020 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 05:14:29.958719 24020 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1782018616648578-3598-0/minicluster-data/master-0/data/instance:
uuid: "79b83bb34fa94b40aaa05fa99d1f8f6c"
format_stamp: "Formatted at 2026-06-21 05:14:29 on dist-test-slave-vrq1"
server_key: "5565632410a1a16dd69fb15c0744a6af"
server_key_iv: "7a13380b2976abeb65126cdabc8786b1"
server_key_version: "encryptionkey@0"
I20260621 05:14:29.959131 24020 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1782018616648578-3598-0/minicluster-data/master-0/wal/instance:
uuid: "79b83bb34fa94b40aaa05fa99d1f8f6c"
format_stamp: "Formatted at 2026-06-21 05:14:29 on dist-test-slave-vrq1"
server_key: "5565632410a1a16dd69fb15c0744a6af"
server_key_iv: "7a13380b2976abeb65126cdabc8786b1"
server_key_version: "encryptionkey@0"
I20260621 05:14:29.960644 24020 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260621 05:14:29.961616 24035 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 05:14:29.961894 24020 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260621 05:14:29.962018 24020 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1782018616648578-3598-0/minicluster-data/master-0/data,/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1782018616648578-3598-0/minicluster-data/master-0/wal
uuid: "79b83bb34fa94b40aaa05fa99d1f8f6c"
format_stamp: "Formatted at 2026-06-21 05:14:29 on dist-test-slave-vrq1"
server_key: "5565632410a1a16dd69fb15c0744a6af"
server_key_iv: "7a13380b2976abeb65126cdabc8786b1"
server_key_version: "encryptionkey@0"
I20260621 05:14:29.962126 24020 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1782018616648578-3598-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1782018616648578-3598-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1782018616648578-3598-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 05:14:29.966017 24020 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 05:14:29.969236 24020 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 05:14:29.969436 24020 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 05:14:29.974746 24020 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:44021
I20260621 05:14:29.974787 24087 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:44021 every 8 connection(s)
I20260621 05:14:29.975193 24020 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1782018616648578-3598-0/minicluster-data/master-0/data/info.pb
I20260621 05:14:29.976341 3598 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWZOx2_/build/release/bin/kudu as pid 24020
I20260621 05:14:29.976392 24088 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 05:14:29.976485 3598 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1782018616648578-3598-0/minicluster-data/master-0/wal/instance
I20260621 05:14:29.976676 3598 external_mini_cluster.cc:1428] Setting key 7f4f490e3a8b8b47fcb59b762d6e8c85
I20260621 05:14:29.979669 24091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:14:29.977454 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54571 (local address 127.0.0.1:44021)
0621 05:14:29.977656 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0621 05:14:29.977664 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0621 05:14:29.977725 (+ 61us) server_negotiation.cc:407] Connection header received
0621 05:14:29.978182 (+ 457us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:14:29.978188 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:14:29.978220 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:14:29.978349 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:14:29.979114 (+ 765us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:14:29.979117 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:14:29.979118 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 05:14:29.979121 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:14:29.979247 (+ 126us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 05:14:29.979269 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:14:29.979327 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0621 05:14:29.979424 (+ 97us) server_negotiation.cc:299] Negotiation successful
0621 05:14:29.979482 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":142,"thread_start_us":75,"threads_started":1}
I20260621 05:14:29.980268 24088 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 79b83bb34fa94b40aaa05fa99d1f8f6c: Bootstrap starting.
I20260621 05:14:29.980809 3598 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260621 05:14:29.981055 24088 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 79b83bb34fa94b40aaa05fa99d1f8f6c: Neither blocks nor log segments found. Creating new log.
I20260621 05:14:29.981346 24088 log.cc:824] T 00000000000000000000000000000000 P 79b83bb34fa94b40aaa05fa99d1f8f6c: Log is configured to *not* fsync() on all Append() calls
I20260621 05:14:29.982081 24088 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 79b83bb34fa94b40aaa05fa99d1f8f6c: No bootstrap required, opened a new log
I20260621 05:14:29.983327 24091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:14:29.982196 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33305 (local address 127.0.0.1:44021)
0621 05:14:29.982393 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0621 05:14:29.982395 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:14:29.982402 (+ 7us) server_negotiation.cc:407] Connection header received
0621 05:14:29.982421 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:14:29.982422 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:14:29.982425 (+ 3us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:14:29.982469 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:14:29.983031 (+ 562us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:14:29.983032 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:14:29.983033 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 05:14:29.983036 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:14:29.983049 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 05:14:29.983052 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:14:29.983098 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0621 05:14:29.983229 (+ 131us) server_negotiation.cc:299] Negotiation successful
0621 05:14:29.983244 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":178}
I20260621 05:14:29.984521 24088 raft_consensus.cc:348] T 00000000000000000000000000000000 P 79b83bb34fa94b40aaa05fa99d1f8f6c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "79b83bb34fa94b40aaa05fa99d1f8f6c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44021 } }
I20260621 05:14:29.984665 24088 raft_consensus.cc:374] T 00000000000000000000000000000000 P 79b83bb34fa94b40aaa05fa99d1f8f6c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 05:14:29.984696 24088 raft_consensus.cc:729] T 00000000000000000000000000000000 P 79b83bb34fa94b40aaa05fa99d1f8f6c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 79b83bb34fa94b40aaa05fa99d1f8f6c, State: Initialized, Role: FOLLOWER
I20260621 05:14:29.984817 24088 consensus_queue.cc:260] T 00000000000000000000000000000000 P 79b83bb34fa94b40aaa05fa99d1f8f6c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "79b83bb34fa94b40aaa05fa99d1f8f6c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44021 } }
I20260621 05:14:29.984912 24088 raft_consensus.cc:388] T 00000000000000000000000000000000 P 79b83bb34fa94b40aaa05fa99d1f8f6c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260621 05:14:29.984935 24088 raft_consensus.cc:482] T 00000000000000000000000000000000 P 79b83bb34fa94b40aaa05fa99d1f8f6c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260621 05:14:29.984966 24088 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 79b83bb34fa94b40aaa05fa99d1f8f6c [term 0 FOLLOWER]: Advancing to term 1
I20260621 05:14:29.985497 24088 raft_consensus.cc:504] T 00000000000000000000000000000000 P 79b83bb34fa94b40aaa05fa99d1f8f6c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "79b83bb34fa94b40aaa05fa99d1f8f6c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44021 } }
I20260621 05:14:29.985589 24088 leader_election.cc:302] T 00000000000000000000000000000000 P 79b83bb34fa94b40aaa05fa99d1f8f6c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 79b83bb34fa94b40aaa05fa99d1f8f6c; no voters:
I20260621 05:14:29.985741 24088 leader_election.cc:288] T 00000000000000000000000000000000 P 79b83bb34fa94b40aaa05fa99d1f8f6c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260621 05:14:29.985842 24098 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 79b83bb34fa94b40aaa05fa99d1f8f6c [term 1 FOLLOWER]: Leader election won for term 1
I20260621 05:14:29.985993 24088 sys_catalog.cc:564] T 00000000000000000000000000000000 P 79b83bb34fa94b40aaa05fa99d1f8f6c [sys.catalog]: configured and running, proceeding with master startup.
I20260621 05:14:29.986007 24098 raft_consensus.cc:686] T 00000000000000000000000000000000 P 79b83bb34fa94b40aaa05fa99d1f8f6c [term 1 LEADER]: Becoming Leader. State: Replica: 79b83bb34fa94b40aaa05fa99d1f8f6c, State: Running, Role: LEADER
I20260621 05:14:29.986124 24098 consensus_queue.cc:237] T 00000000000000000000000000000000 P 79b83bb34fa94b40aaa05fa99d1f8f6c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "79b83bb34fa94b40aaa05fa99d1f8f6c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44021 } }
I20260621 05:14:29.986490 24099 sys_catalog.cc:455] T 00000000000000000000000000000000 P 79b83bb34fa94b40aaa05fa99d1f8f6c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "79b83bb34fa94b40aaa05fa99d1f8f6c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "79b83bb34fa94b40aaa05fa99d1f8f6c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44021 } } }
I20260621 05:14:29.986512 24100 sys_catalog.cc:455] T 00000000000000000000000000000000 P 79b83bb34fa94b40aaa05fa99d1f8f6c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 79b83bb34fa94b40aaa05fa99d1f8f6c. Latest consensus state: current_term: 1 leader_uuid: "79b83bb34fa94b40aaa05fa99d1f8f6c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "79b83bb34fa94b40aaa05fa99d1f8f6c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44021 } } }
I20260621 05:14:29.986707 24099 sys_catalog.cc:458] T 00000000000000000000000000000000 P 79b83bb34fa94b40aaa05fa99d1f8f6c [sys.catalog]: This master's current role is: LEADER
I20260621 05:14:29.986773 24100 sys_catalog.cc:458] T 00000000000000000000000000000000 P 79b83bb34fa94b40aaa05fa99d1f8f6c [sys.catalog]: This master's current role is: LEADER
I20260621 05:14:29.987358 24107 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260621 05:14:29.987931 24107 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260621 05:14:29.989764 24107 catalog_manager.cc:1347] Generated new cluster ID: 28fc155320504830b234685ea749a63e
I20260621 05:14:29.989821 24107 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260621 05:14:30.006657 24107 catalog_manager.cc:1370] Generated new certificate authority record
I20260621 05:14:30.006880 24107 catalog_manager.cc:1504] Loading token signing keys...
I20260621 05:14:30.014178 24107 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 79b83bb34fa94b40aaa05fa99d1f8f6c: Generated new TSK 0
I20260621 05:14:30.014392 24107 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260621 05:14:30.023530 3598 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWZOx2_/build/release/bin/kudu with pid 24020
2026-06-21T05:14:30Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1371 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1782018616648578-3598-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 21 05:14:30 dist-test-slave-vrq1 krb5kdc[24119](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 21 05:14:30 dist-test-slave-vrq1 krb5kdc[24119](info): set up 2 sockets
Jun 21 05:14:30 dist-test-slave-vrq1 krb5kdc[24119](info): commencing operation
krb5kdc: starting...
W20260621 05:14:31.067636 3598 mini_kdc.cc:121] Time spent starting KDC: real 1.025s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1782018616648578-3598-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1782018616648578-3598-0/krb5kdc/test-user.keytab.
Jun 21 05:14:31 dist-test-slave-vrq1 krb5kdc[24119](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782018871, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-21T05:14:31Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-21T05:14:31Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1782018616648578-3598-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1782018616648578-3598-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260621 05:14:31.195425 3598 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWZOx2_/build/release/bin/kudu
/tmp/dist-test-taskWZOx2_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1782018616648578-3598-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1782018616648578-3598-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1782018616648578-3598-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1782018616648578-3598-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46627
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1782018616648578-3598-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:33617
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:46627
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1782018616648578-3598-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1782018616648578-3598-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1782018616648578-3598-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 05:14:31.280681 24132 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 05:14:31.280891 24132 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 05:14:31.280915 24132 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 05:14:31.282097 24132 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 05:14:31.282141 24132 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 05:14:31.282155 24132 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 05:14:31.282168 24132 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 05:14:31.282181 24132 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 05:14:31.283761 24132 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:33617
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1782018616648578-3598-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1782018616648578-3598-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:46627
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1782018616648578-3598-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46627
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1782018616648578-3598-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24132
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1782018616648578-3598-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 21 Jun 2026 04:16:16 UTC on ccycloud.quasar-xycaha.root.comops.site
build id 4457881
I20260621 05:14:31.284032 24132 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 05:14:31.284286 24132 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 05:14:31.289368 24137 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:14:31.289404 24138 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:14:31.289425 24140 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:14:31.289829 24132 server_base.cc:1034] running on GCE node
I20260621 05:14:31.290095 24132 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 05:14:31.290473 24132 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 05:14:31.291674 24132 hybrid_clock.cc:648] HybridClock initialized: now 1782018871291658 us; error 29 us; skew 500 ppm
Jun 21 05:14:31 dist-test-slave-vrq1 krb5kdc[24119](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782018871, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 05:14:31.295017 24132 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260621 05:14:31.295718 24132 webserver.cc:460] Webserver started at http://127.0.0.1:39149/ using document root <none> and password file <none>
I20260621 05:14:31.296085 24132 fs_manager.cc:359] Metadata directory not provided
I20260621 05:14:31.296173 24132 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 05:14:31.296336 24132 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 05:14:31.297957 24132 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1782018616648578-3598-0/minicluster-data/master-0/data/instance:
uuid: "3b3dd0cbda7741aa8ed72b657210183c"
format_stamp: "Formatted at 2026-06-21 05:14:31 on dist-test-slave-vrq1"
server_key: "d22003c12e1d91dfa9b7a75fc8441ae9"
server_key_iv: "9c0116aaa7adee88fb91a91e33882c08"
server_key_version: "encryptionkey@0"
I20260621 05:14:31.298477 24132 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1782018616648578-3598-0/minicluster-data/master-0/wal/instance:
uuid: "3b3dd0cbda7741aa8ed72b657210183c"
format_stamp: "Formatted at 2026-06-21 05:14:31 on dist-test-slave-vrq1"
server_key: "d22003c12e1d91dfa9b7a75fc8441ae9"
server_key_iv: "9c0116aaa7adee88fb91a91e33882c08"
server_key_version: "encryptionkey@0"
I20260621 05:14:31.300475 24132 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.002s sys 0.000s
I20260621 05:14:31.301628 24147 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 05:14:31.301941 24132 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260621 05:14:31.302053 24132 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1782018616648578-3598-0/minicluster-data/master-0/data,/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1782018616648578-3598-0/minicluster-data/master-0/wal
uuid: "3b3dd0cbda7741aa8ed72b657210183c"
format_stamp: "Formatted at 2026-06-21 05:14:31 on dist-test-slave-vrq1"
server_key: "d22003c12e1d91dfa9b7a75fc8441ae9"
server_key_iv: "9c0116aaa7adee88fb91a91e33882c08"
server_key_version: "encryptionkey@0"
I20260621 05:14:31.302143 24132 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1782018616648578-3598-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1782018616648578-3598-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1782018616648578-3598-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 05:14:31.321606 24132 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 05:14:31.324219 24132 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 05:14:31.324361 24132 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 05:14:31.328352 24132 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:46627
I20260621 05:14:31.328398 24199 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:46627 every 8 connection(s)
I20260621 05:14:31.328797 24132 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1782018616648578-3598-0/minicluster-data/master-0/data/info.pb
I20260621 05:14:31.329811 24200 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 05:14:31.330894 3598 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWZOx2_/build/release/bin/kudu as pid 24132
I20260621 05:14:31.331022 3598 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1782018616648578-3598-0/minicluster-data/master-0/wal/instance
I20260621 05:14:31.331244 3598 external_mini_cluster.cc:1428] Setting key f80a29eb0437bbf5839d8d75e26e30c3
I20260621 05:14:31.332720 24200 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3b3dd0cbda7741aa8ed72b657210183c: Bootstrap starting.
I20260621 05:14:31.333321 24200 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3b3dd0cbda7741aa8ed72b657210183c: Neither blocks nor log segments found. Creating new log.
I20260621 05:14:31.333616 24200 log.cc:824] T 00000000000000000000000000000000 P 3b3dd0cbda7741aa8ed72b657210183c: Log is configured to *not* fsync() on all Append() calls
I20260621 05:14:31.334323 24200 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3b3dd0cbda7741aa8ed72b657210183c: No bootstrap required, opened a new log
I20260621 05:14:31.336247 24200 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3b3dd0cbda7741aa8ed72b657210183c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b3dd0cbda7741aa8ed72b657210183c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46627 } }
I20260621 05:14:31.336408 24200 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3b3dd0cbda7741aa8ed72b657210183c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 05:14:31.336455 24200 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3b3dd0cbda7741aa8ed72b657210183c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3b3dd0cbda7741aa8ed72b657210183c, State: Initialized, Role: FOLLOWER
I20260621 05:14:31.336602 24200 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3b3dd0cbda7741aa8ed72b657210183c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b3dd0cbda7741aa8ed72b657210183c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46627 } }
I20260621 05:14:31.336709 24200 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3b3dd0cbda7741aa8ed72b657210183c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260621 05:14:31.336751 24200 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3b3dd0cbda7741aa8ed72b657210183c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260621 05:14:31.336802 24200 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3b3dd0cbda7741aa8ed72b657210183c [term 0 FOLLOWER]: Advancing to term 1
I20260621 05:14:31.337677 24200 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3b3dd0cbda7741aa8ed72b657210183c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b3dd0cbda7741aa8ed72b657210183c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46627 } }
I20260621 05:14:31.337802 24200 leader_election.cc:302] T 00000000000000000000000000000000 P 3b3dd0cbda7741aa8ed72b657210183c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3b3dd0cbda7741aa8ed72b657210183c; no voters:
I20260621 05:14:31.338112 24200 leader_election.cc:288] T 00000000000000000000000000000000 P 3b3dd0cbda7741aa8ed72b657210183c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260621 05:14:31.338135 24205 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3b3dd0cbda7741aa8ed72b657210183c [term 1 FOLLOWER]: Leader election won for term 1
Jun 21 05:14:31 dist-test-slave-vrq1 krb5kdc[24119](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782018871, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260621 05:14:31.338294 24205 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3b3dd0cbda7741aa8ed72b657210183c [term 1 LEADER]: Becoming Leader. State: Replica: 3b3dd0cbda7741aa8ed72b657210183c, State: Running, Role: LEADER
I20260621 05:14:31.338371 24205 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3b3dd0cbda7741aa8ed72b657210183c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b3dd0cbda7741aa8ed72b657210183c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46627 } }
I20260621 05:14:31.338438 24200 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3b3dd0cbda7741aa8ed72b657210183c [sys.catalog]: configured and running, proceeding with master startup.
I20260621 05:14:31.338725 24207 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3b3dd0cbda7741aa8ed72b657210183c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3b3dd0cbda7741aa8ed72b657210183c. Latest consensus state: current_term: 1 leader_uuid: "3b3dd0cbda7741aa8ed72b657210183c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b3dd0cbda7741aa8ed72b657210183c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46627 } } }
I20260621 05:14:31.338963 24207 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3b3dd0cbda7741aa8ed72b657210183c [sys.catalog]: This master's current role is: LEADER
I20260621 05:14:31.338886 24206 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3b3dd0cbda7741aa8ed72b657210183c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3b3dd0cbda7741aa8ed72b657210183c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b3dd0cbda7741aa8ed72b657210183c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46627 } } }
I20260621 05:14:31.339151 24206 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3b3dd0cbda7741aa8ed72b657210183c [sys.catalog]: This master's current role is: LEADER
I20260621 05:14:31.339933 24214 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260621 05:14:31.340835 24214 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260621 05:14:31.342531 24214 catalog_manager.cc:1347] Generated new cluster ID: 99f2c2228ea14c8683fc87a2ede96dcb
I20260621 05:14:31.342583 24214 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260621 05:14:31.343466 24203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:14:31.331804 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34131 (local address 127.0.0.1:46627)
0621 05:14:31.332018 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0621 05:14:31.332034 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0621 05:14:31.332061 (+ 27us) server_negotiation.cc:407] Connection header received
0621 05:14:31.332517 (+ 456us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:14:31.332526 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:14:31.332584 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:14:31.332732 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:14:31.333371 (+ 639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:14:31.334371 (+ 1000us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:14:31.335454 (+ 1083us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:14:31.335907 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:14:31.336040 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:14:31.338889 (+ 2849us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:14:31.338894 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:14:31.338894 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:14:31.338909 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:14:31.341269 (+ 2360us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:14:31.341740 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:14:31.341744 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:14:31.341745 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:14:31.341858 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:14:31.342074 (+ 216us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:14:31.342075 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:14:31.342075 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:14:31.342381 (+ 306us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:14:31.342525 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0621 05:14:31.343066 (+ 541us) server_negotiation.cc:299] Negotiation successful
0621 05:14:31.343261 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":154,"thread_start_us":98,"threads_started":1}
I20260621 05:14:31.344416 3598 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260621 05:14:31.353322 24203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:14:31.345909 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57041 (local address 127.0.0.1:46627)
0621 05:14:31.345945 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0621 05:14:31.345947 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:14:31.346023 (+ 76us) server_negotiation.cc:407] Connection header received
0621 05:14:31.346085 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:14:31.346086 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:14:31.346096 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:14:31.346163 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:14:31.346774 (+ 611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:14:31.347661 (+ 887us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:14:31.348657 (+ 996us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:14:31.348991 (+ 334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:14:31.349070 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:14:31.350466 (+ 1396us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:14:31.350468 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:14:31.350468 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:14:31.350476 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:14:31.352142 (+ 1666us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:14:31.352529 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:14:31.352530 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:14:31.352530 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:14:31.352594 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:14:31.352799 (+ 205us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:14:31.352799 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:14:31.352800 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:14:31.352936 (+ 136us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:14:31.353036 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0621 05:14:31.353143 (+ 107us) server_negotiation.cc:299] Negotiation successful
0621 05:14:31.353222 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260621 05:14:31.354173 24214 catalog_manager.cc:1370] Generated new certificate authority record
I20260621 05:14:31.354698 24214 catalog_manager.cc:1504] Loading token signing keys...
I20260621 05:14:31.358573 24214 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3b3dd0cbda7741aa8ed72b657210183c: Generated new TSK 0
I20260621 05:14:31.358773 24214 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260621 05:14:31.367815 3598 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWZOx2_/build/release/bin/kudu with pid 24132
2026-06-21T05:14:31Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (1343 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1782018616648578-3598-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 21 05:14:31 dist-test-slave-vrq1 krb5kdc[24231](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 21 05:14:31 dist-test-slave-vrq1 krb5kdc[24231](info): set up 2 sockets
Jun 21 05:14:31 dist-test-slave-vrq1 krb5kdc[24231](info): commencing operation
krb5kdc: starting...
W20260621 05:14:32.423805 3598 mini_kdc.cc:121] Time spent starting KDC: real 1.037s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1782018616648578-3598-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1782018616648578-3598-0/krb5kdc/test-user.keytab.
Jun 21 05:14:32 dist-test-slave-vrq1 krb5kdc[24231](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782018872, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-21T05:14:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-21T05:14:32Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1782018616648578-3598-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1782018616648578-3598-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260621 05:14:32.556658 3598 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWZOx2_/build/release/bin/kudu
/tmp/dist-test-taskWZOx2_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1782018616648578-3598-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1782018616648578-3598-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1782018616648578-3598-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1782018616648578-3598-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:32985
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1782018616648578-3598-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:40653
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:32985
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1782018616648578-3598-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1782018616648578-3598-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1782018616648578-3598-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 05:14:32.643554 24244 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 05:14:32.643810 24244 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 05:14:32.643853 24244 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 05:14:32.645116 24244 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 05:14:32.645201 24244 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 05:14:32.645226 24244 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 05:14:32.645248 24244 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 05:14:32.645277 24244 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 05:14:32.647670 24244 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:40653
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1782018616648578-3598-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1782018616648578-3598-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:32985
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1782018616648578-3598-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:32985
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1782018616648578-3598-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24244
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1782018616648578-3598-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 21 Jun 2026 04:16:16 UTC on ccycloud.quasar-xycaha.root.comops.site
build id 4457881
I20260621 05:14:32.647996 24244 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 05:14:32.648265 24244 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 05:14:32.653609 24252 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:14:32.653591 24249 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:14:32.653594 24250 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:14:32.654049 24244 server_base.cc:1034] running on GCE node
I20260621 05:14:32.654331 24244 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 05:14:32.654702 24244 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 05:14:32.655889 24244 hybrid_clock.cc:648] HybridClock initialized: now 1782018872655863 us; error 43 us; skew 500 ppm
Jun 21 05:14:32 dist-test-slave-vrq1 krb5kdc[24231](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782018872, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 05:14:32.665918 24244 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260621 05:14:32.666517 24244 webserver.cc:460] Webserver started at http://127.0.0.1:42575/ using document root <none> and password file <none>
I20260621 05:14:32.666827 24244 fs_manager.cc:359] Metadata directory not provided
I20260621 05:14:32.666908 24244 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 05:14:32.667107 24244 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 05:14:32.668375 24244 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1782018616648578-3598-0/minicluster-data/master-0/data/instance:
uuid: "b2f287c929044ce99d88674c8a0fac44"
format_stamp: "Formatted at 2026-06-21 05:14:32 on dist-test-slave-vrq1"
server_key: "fe0fea8db902b15d42cf44d5b9dda276"
server_key_iv: "1023604fb81599cdadad1482476be17a"
server_key_version: "encryptionkey@0"
I20260621 05:14:32.668788 24244 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1782018616648578-3598-0/minicluster-data/master-0/wal/instance:
uuid: "b2f287c929044ce99d88674c8a0fac44"
format_stamp: "Formatted at 2026-06-21 05:14:32 on dist-test-slave-vrq1"
server_key: "fe0fea8db902b15d42cf44d5b9dda276"
server_key_iv: "1023604fb81599cdadad1482476be17a"
server_key_version: "encryptionkey@0"
I20260621 05:14:32.670303 24244 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260621 05:14:32.671135 24259 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 05:14:32.671334 24244 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260621 05:14:32.671445 24244 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1782018616648578-3598-0/minicluster-data/master-0/data,/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1782018616648578-3598-0/minicluster-data/master-0/wal
uuid: "b2f287c929044ce99d88674c8a0fac44"
format_stamp: "Formatted at 2026-06-21 05:14:32 on dist-test-slave-vrq1"
server_key: "fe0fea8db902b15d42cf44d5b9dda276"
server_key_iv: "1023604fb81599cdadad1482476be17a"
server_key_version: "encryptionkey@0"
I20260621 05:14:32.671533 24244 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1782018616648578-3598-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1782018616648578-3598-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1782018616648578-3598-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 05:14:32.684767 24244 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 05:14:32.687090 24244 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 05:14:32.687342 24244 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 05:14:32.691912 24244 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:32985
I20260621 05:14:32.691965 24311 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:32985 every 8 connection(s)
I20260621 05:14:32.692480 24244 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1782018616648578-3598-0/minicluster-data/master-0/data/info.pb
I20260621 05:14:32.692782 3598 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWZOx2_/build/release/bin/kudu as pid 24244
I20260621 05:14:32.692927 3598 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1782018616648578-3598-0/minicluster-data/master-0/wal/instance
I20260621 05:14:32.693148 3598 external_mini_cluster.cc:1428] Setting key d425c0a793289b7768e56eff93f7885c
I20260621 05:14:32.693831 24312 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 05:14:32.696630 24312 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b2f287c929044ce99d88674c8a0fac44: Bootstrap starting.
I20260621 05:14:32.697284 24312 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b2f287c929044ce99d88674c8a0fac44: Neither blocks nor log segments found. Creating new log.
I20260621 05:14:32.697642 24312 log.cc:824] T 00000000000000000000000000000000 P b2f287c929044ce99d88674c8a0fac44: Log is configured to *not* fsync() on all Append() calls
I20260621 05:14:32.698592 24312 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b2f287c929044ce99d88674c8a0fac44: No bootstrap required, opened a new log
I20260621 05:14:32.699164 24315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:14:32.693756 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40881 (local address 127.0.0.1:32985)
0621 05:14:32.693985 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0621 05:14:32.693995 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0621 05:14:32.694029 (+ 34us) server_negotiation.cc:407] Connection header received
0621 05:14:32.694492 (+ 463us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:14:32.694499 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:14:32.694547 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:14:32.694704 (+ 157us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:14:32.694936 (+ 232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:14:32.695925 (+ 989us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:14:32.696777 (+ 852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:14:32.697210 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:14:32.697328 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:14:32.698240 (+ 912us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:14:32.698245 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:14:32.698245 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 05:14:32.698465 (+ 220us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:14:32.698507 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 05:14:32.698524 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:14:32.698693 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0621 05:14:32.698886 (+ 193us) server_negotiation.cc:299] Negotiation successful
0621 05:14:32.698964 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":170,"thread_start_us":102,"threads_started":1}
I20260621 05:14:32.701000 3598 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260621 05:14:32.701776 24312 raft_consensus.cc:348] T 00000000000000000000000000000000 P b2f287c929044ce99d88674c8a0fac44 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b2f287c929044ce99d88674c8a0fac44" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32985 } }
I20260621 05:14:32.701996 24312 raft_consensus.cc:374] T 00000000000000000000000000000000 P b2f287c929044ce99d88674c8a0fac44 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 05:14:32.702045 24312 raft_consensus.cc:729] T 00000000000000000000000000000000 P b2f287c929044ce99d88674c8a0fac44 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b2f287c929044ce99d88674c8a0fac44, State: Initialized, Role: FOLLOWER
I20260621 05:14:32.702208 24312 consensus_queue.cc:260] T 00000000000000000000000000000000 P b2f287c929044ce99d88674c8a0fac44 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b2f287c929044ce99d88674c8a0fac44" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32985 } }
I20260621 05:14:32.702332 24312 raft_consensus.cc:388] T 00000000000000000000000000000000 P b2f287c929044ce99d88674c8a0fac44 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260621 05:14:32.702374 24312 raft_consensus.cc:482] T 00000000000000000000000000000000 P b2f287c929044ce99d88674c8a0fac44 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260621 05:14:32.702428 24312 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b2f287c929044ce99d88674c8a0fac44 [term 0 FOLLOWER]: Advancing to term 1
I20260621 05:14:32.703256 24312 raft_consensus.cc:504] T 00000000000000000000000000000000 P b2f287c929044ce99d88674c8a0fac44 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b2f287c929044ce99d88674c8a0fac44" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32985 } }
I20260621 05:14:32.703401 24312 leader_election.cc:302] T 00000000000000000000000000000000 P b2f287c929044ce99d88674c8a0fac44 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b2f287c929044ce99d88674c8a0fac44; no voters:
I20260621 05:14:32.703678 24312 leader_election.cc:288] T 00000000000000000000000000000000 P b2f287c929044ce99d88674c8a0fac44 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260621 05:14:32.703833 24322 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b2f287c929044ce99d88674c8a0fac44 [term 1 FOLLOWER]: Leader election won for term 1
I20260621 05:14:32.704015 24312 sys_catalog.cc:564] T 00000000000000000000000000000000 P b2f287c929044ce99d88674c8a0fac44 [sys.catalog]: configured and running, proceeding with master startup.
I20260621 05:14:32.704061 24322 raft_consensus.cc:686] T 00000000000000000000000000000000 P b2f287c929044ce99d88674c8a0fac44 [term 1 LEADER]: Becoming Leader. State: Replica: b2f287c929044ce99d88674c8a0fac44, State: Running, Role: LEADER
I20260621 05:14:32.704291 24322 consensus_queue.cc:237] T 00000000000000000000000000000000 P b2f287c929044ce99d88674c8a0fac44 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b2f287c929044ce99d88674c8a0fac44" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32985 } }
I20260621 05:14:32.704661 24323 sys_catalog.cc:455] T 00000000000000000000000000000000 P b2f287c929044ce99d88674c8a0fac44 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b2f287c929044ce99d88674c8a0fac44" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b2f287c929044ce99d88674c8a0fac44" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32985 } } }
I20260621 05:14:32.704752 24323 sys_catalog.cc:458] T 00000000000000000000000000000000 P b2f287c929044ce99d88674c8a0fac44 [sys.catalog]: This master's current role is: LEADER
I20260621 05:14:32.704929 24324 sys_catalog.cc:455] T 00000000000000000000000000000000 P b2f287c929044ce99d88674c8a0fac44 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b2f287c929044ce99d88674c8a0fac44. Latest consensus state: current_term: 1 leader_uuid: "b2f287c929044ce99d88674c8a0fac44" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b2f287c929044ce99d88674c8a0fac44" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32985 } } }
I20260621 05:14:32.704988 24324 sys_catalog.cc:458] T 00000000000000000000000000000000 P b2f287c929044ce99d88674c8a0fac44 [sys.catalog]: This master's current role is: LEADER
I20260621 05:14:32.705427 24329 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260621 05:14:32.706019 24329 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260621 05:14:32.707397 24315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:14:32.702474 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60567 (local address 127.0.0.1:32985)
0621 05:14:32.702522 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0621 05:14:32.702524 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:14:32.703447 (+ 923us) server_negotiation.cc:407] Connection header received
0621 05:14:32.703506 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:14:32.703507 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:14:32.703522 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:14:32.703597 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:14:32.703844 (+ 247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:14:32.704774 (+ 930us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:14:32.705969 (+ 1195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:14:32.706253 (+ 284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:14:32.706334 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:14:32.707098 (+ 764us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:14:32.707099 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:14:32.707100 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 05:14:32.707105 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:14:32.707121 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 05:14:32.707126 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:14:32.707216 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0621 05:14:32.707285 (+ 69us) server_negotiation.cc:299] Negotiation successful
0621 05:14:32.707315 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260621 05:14:32.708110 24329 catalog_manager.cc:1347] Generated new cluster ID: 16a458427f254a87a6c51c429518cad0
I20260621 05:14:32.708153 24329 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260621 05:14:32.736438 24329 catalog_manager.cc:1370] Generated new certificate authority record
I20260621 05:14:32.737422 24329 catalog_manager.cc:1504] Loading token signing keys...
I20260621 05:14:32.743374 24329 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b2f287c929044ce99d88674c8a0fac44: Generated new TSK 0
I20260621 05:14:32.743731 24329 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260621 05:14:32.754686 3598 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWZOx2_/build/release/bin/kudu with pid 24244
2026-06-21T05:14:32Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1386 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1782018616648578-3598-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 21 05:14:32 dist-test-slave-vrq1 krb5kdc[24343](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 21 05:14:32 dist-test-slave-vrq1 krb5kdc[24343](info): set up 2 sockets
Jun 21 05:14:32 dist-test-slave-vrq1 krb5kdc[24343](info): commencing operation
krb5kdc: starting...
W20260621 05:14:33.807574 3598 mini_kdc.cc:121] Time spent starting KDC: real 1.034s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1782018616648578-3598-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1782018616648578-3598-0/krb5kdc/test-user.keytab.
Jun 21 05:14:33 dist-test-slave-vrq1 krb5kdc[24343](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782018873, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-21T05:14:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-21T05:14:33Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1782018616648578-3598-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1782018616648578-3598-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260621 05:14:33.939254 3598 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWZOx2_/build/release/bin/kudu
/tmp/dist-test-taskWZOx2_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1782018616648578-3598-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1782018616648578-3598-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1782018616648578-3598-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1782018616648578-3598-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43855
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1782018616648578-3598-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:41643
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:43855
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1782018616648578-3598-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1782018616648578-3598-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1782018616648578-3598-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 05:14:34.024727 24356 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 05:14:34.024966 24356 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 05:14:34.024991 24356 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 05:14:34.026320 24356 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 05:14:34.026374 24356 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 05:14:34.026391 24356 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 05:14:34.026403 24356 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 05:14:34.026417 24356 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 05:14:34.028146 24356 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:41643
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1782018616648578-3598-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1782018616648578-3598-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:43855
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1782018616648578-3598-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43855
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1782018616648578-3598-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24356
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1782018616648578-3598-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 21 Jun 2026 04:16:16 UTC on ccycloud.quasar-xycaha.root.comops.site
build id 4457881
I20260621 05:14:34.028434 24356 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 05:14:34.028688 24356 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 05:14:34.033913 24364 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:14:34.033996 24356 server_base.cc:1034] running on GCE node
W20260621 05:14:34.033931 24362 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:14:34.033913 24361 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:14:34.034401 24356 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 05:14:34.034662 24356 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 05:14:34.035842 24356 hybrid_clock.cc:648] HybridClock initialized: now 1782018874035817 us; error 38 us; skew 500 ppm
Jun 21 05:14:34 dist-test-slave-vrq1 krb5kdc[24343](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782018874, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 05:14:34.038378 24356 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260621 05:14:34.038805 24356 webserver.cc:460] Webserver started at http://127.0.0.1:43175/ using document root <none> and password file <none>
I20260621 05:14:34.038995 24356 fs_manager.cc:359] Metadata directory not provided
I20260621 05:14:34.039038 24356 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 05:14:34.039129 24356 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 05:14:34.040274 24356 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1782018616648578-3598-0/minicluster-data/master-0/data/instance:
uuid: "7305c42ca7c94762a3f6104a3df60773"
format_stamp: "Formatted at 2026-06-21 05:14:34 on dist-test-slave-vrq1"
server_key: "c2107f2390b48ef1420becce434de6ea"
server_key_iv: "ffe28f6b12ad058a730a5bc7d7dcf66e"
server_key_version: "encryptionkey@0"
I20260621 05:14:34.040603 24356 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1782018616648578-3598-0/minicluster-data/master-0/wal/instance:
uuid: "7305c42ca7c94762a3f6104a3df60773"
format_stamp: "Formatted at 2026-06-21 05:14:34 on dist-test-slave-vrq1"
server_key: "c2107f2390b48ef1420becce434de6ea"
server_key_iv: "ffe28f6b12ad058a730a5bc7d7dcf66e"
server_key_version: "encryptionkey@0"
I20260621 05:14:34.041908 24356 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260621 05:14:34.042829 24371 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 05:14:34.043066 24356 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260621 05:14:34.043228 24356 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1782018616648578-3598-0/minicluster-data/master-0/data,/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1782018616648578-3598-0/minicluster-data/master-0/wal
uuid: "7305c42ca7c94762a3f6104a3df60773"
format_stamp: "Formatted at 2026-06-21 05:14:34 on dist-test-slave-vrq1"
server_key: "c2107f2390b48ef1420becce434de6ea"
server_key_iv: "ffe28f6b12ad058a730a5bc7d7dcf66e"
server_key_version: "encryptionkey@0"
I20260621 05:14:34.043321 24356 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1782018616648578-3598-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1782018616648578-3598-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1782018616648578-3598-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 05:14:34.046929 24356 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 05:14:34.048938 24356 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 05:14:34.049104 24356 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 05:14:34.053231 24356 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:43855
I20260621 05:14:34.053263 24423 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:43855 every 8 connection(s)
I20260621 05:14:34.053615 24356 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1782018616648578-3598-0/minicluster-data/master-0/data/info.pb
I20260621 05:14:34.054562 24424 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 05:14:34.054576 3598 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWZOx2_/build/release/bin/kudu as pid 24356
I20260621 05:14:34.054766 3598 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1782018616648578-3598-0/minicluster-data/master-0/wal/instance
I20260621 05:14:34.054967 3598 external_mini_cluster.cc:1428] Setting key e83a5509ba9ea4db6821c6e46967ccc0
I20260621 05:14:34.057447 24427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:14:34.055553 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43039 (local address 127.0.0.1:43855)
0621 05:14:34.055728 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0621 05:14:34.055735 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0621 05:14:34.055760 (+ 25us) server_negotiation.cc:407] Connection header received
0621 05:14:34.056234 (+ 474us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:14:34.056241 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:14:34.056267 (+ 26us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:14:34.056382 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:14:34.057003 (+ 621us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:14:34.057007 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:14:34.057007 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 05:14:34.057010 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:14:34.057045 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 05:14:34.057055 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:14:34.057108 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0621 05:14:34.057194 (+ 86us) server_negotiation.cc:299] Negotiation successful
0621 05:14:34.057225 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":119,"thread_start_us":95,"threads_started":1}
I20260621 05:14:34.058272 24424 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7305c42ca7c94762a3f6104a3df60773: Bootstrap starting.
I20260621 05:14:34.058607 3598 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260621 05:14:34.058945 24424 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7305c42ca7c94762a3f6104a3df60773: Neither blocks nor log segments found. Creating new log.
I20260621 05:14:34.059288 24424 log.cc:824] T 00000000000000000000000000000000 P 7305c42ca7c94762a3f6104a3df60773: Log is configured to *not* fsync() on all Append() calls
I20260621 05:14:34.060367 24424 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7305c42ca7c94762a3f6104a3df60773: No bootstrap required, opened a new log
I20260621 05:14:34.061509 24427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:14:34.060259 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47981 (local address 127.0.0.1:43855)
0621 05:14:34.060296 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0621 05:14:34.060298 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:14:34.060436 (+ 138us) server_negotiation.cc:407] Connection header received
0621 05:14:34.060544 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:14:34.060545 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:14:34.060550 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:14:34.060616 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:14:34.061273 (+ 657us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:14:34.061275 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:14:34.061275 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 05:14:34.061279 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:14:34.061293 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 05:14:34.061297 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:14:34.061354 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0621 05:14:34.061412 (+ 58us) server_negotiation.cc:299] Negotiation successful
0621 05:14:34.061428 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260621 05:14:34.062516 24424 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7305c42ca7c94762a3f6104a3df60773 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7305c42ca7c94762a3f6104a3df60773" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43855 } }
I20260621 05:14:34.062686 24424 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7305c42ca7c94762a3f6104a3df60773 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 05:14:34.062745 24424 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7305c42ca7c94762a3f6104a3df60773 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7305c42ca7c94762a3f6104a3df60773, State: Initialized, Role: FOLLOWER
I20260621 05:14:34.062878 24424 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7305c42ca7c94762a3f6104a3df60773 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7305c42ca7c94762a3f6104a3df60773" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43855 } }
I20260621 05:14:34.062973 24424 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7305c42ca7c94762a3f6104a3df60773 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260621 05:14:34.063019 24424 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7305c42ca7c94762a3f6104a3df60773 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260621 05:14:34.063067 24424 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7305c42ca7c94762a3f6104a3df60773 [term 0 FOLLOWER]: Advancing to term 1
I20260621 05:14:34.063658 24424 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7305c42ca7c94762a3f6104a3df60773 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7305c42ca7c94762a3f6104a3df60773" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43855 } }
I20260621 05:14:34.063782 24424 leader_election.cc:302] T 00000000000000000000000000000000 P 7305c42ca7c94762a3f6104a3df60773 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7305c42ca7c94762a3f6104a3df60773; no voters:
I20260621 05:14:34.063987 24424 leader_election.cc:288] T 00000000000000000000000000000000 P 7305c42ca7c94762a3f6104a3df60773 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260621 05:14:34.064042 24434 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7305c42ca7c94762a3f6104a3df60773 [term 1 FOLLOWER]: Leader election won for term 1
I20260621 05:14:34.064236 24424 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7305c42ca7c94762a3f6104a3df60773 [sys.catalog]: configured and running, proceeding with master startup.
I20260621 05:14:34.064270 24434 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7305c42ca7c94762a3f6104a3df60773 [term 1 LEADER]: Becoming Leader. State: Replica: 7305c42ca7c94762a3f6104a3df60773, State: Running, Role: LEADER
I20260621 05:14:34.064409 24434 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7305c42ca7c94762a3f6104a3df60773 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7305c42ca7c94762a3f6104a3df60773" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43855 } }
I20260621 05:14:34.064808 24435 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7305c42ca7c94762a3f6104a3df60773 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7305c42ca7c94762a3f6104a3df60773" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7305c42ca7c94762a3f6104a3df60773" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43855 } } }
I20260621 05:14:34.064836 24436 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7305c42ca7c94762a3f6104a3df60773 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7305c42ca7c94762a3f6104a3df60773. Latest consensus state: current_term: 1 leader_uuid: "7305c42ca7c94762a3f6104a3df60773" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7305c42ca7c94762a3f6104a3df60773" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43855 } } }
I20260621 05:14:34.064996 24436 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7305c42ca7c94762a3f6104a3df60773 [sys.catalog]: This master's current role is: LEADER
I20260621 05:14:34.064983 24435 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7305c42ca7c94762a3f6104a3df60773 [sys.catalog]: This master's current role is: LEADER
I20260621 05:14:34.065425 24443 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260621 05:14:34.065829 24443 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260621 05:14:34.067517 24443 catalog_manager.cc:1347] Generated new cluster ID: ab0f7fa14ab74273b88f72d5d49a5550
I20260621 05:14:34.067575 24443 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260621 05:14:34.085206 24443 catalog_manager.cc:1370] Generated new certificate authority record
I20260621 05:14:34.085402 24443 catalog_manager.cc:1504] Loading token signing keys...
I20260621 05:14:34.089620 24443 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7305c42ca7c94762a3f6104a3df60773: Generated new TSK 0
I20260621 05:14:34.089846 24443 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260621 05:14:34.093580 3598 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWZOx2_/build/release/bin/kudu with pid 24356
2026-06-21T05:14:34Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1338 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1782018616648578-3598-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 21 05:14:34 dist-test-slave-vrq1 krb5kdc[24455](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 21 05:14:34 dist-test-slave-vrq1 krb5kdc[24455](info): set up 2 sockets
Jun 21 05:14:34 dist-test-slave-vrq1 krb5kdc[24455](info): commencing operation
krb5kdc: starting...
W20260621 05:14:35.151854 3598 mini_kdc.cc:121] Time spent starting KDC: real 1.040s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1782018616648578-3598-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1782018616648578-3598-0/krb5kdc/test-user.keytab.
Jun 21 05:14:35 dist-test-slave-vrq1 krb5kdc[24455](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782018875, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-21T05:14:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-21T05:14:35Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1782018616648578-3598-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1782018616648578-3598-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260621 05:14:35.286895 3598 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWZOx2_/build/release/bin/kudu
/tmp/dist-test-taskWZOx2_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1782018616648578-3598-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1782018616648578-3598-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1782018616648578-3598-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1782018616648578-3598-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33163
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1782018616648578-3598-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34027
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:33163
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1782018616648578-3598-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1782018616648578-3598-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1782018616648578-3598-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 05:14:35.378902 24468 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 05:14:35.379274 24468 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 05:14:35.379356 24468 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 05:14:35.380812 24468 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 05:14:35.380903 24468 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 05:14:35.380928 24468 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 05:14:35.380951 24468 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 05:14:35.380978 24468 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 05:14:35.383389 24468 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34027
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1782018616648578-3598-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1782018616648578-3598-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:33163
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1782018616648578-3598-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33163
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1782018616648578-3598-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24468
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1782018616648578-3598-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 21 Jun 2026 04:16:16 UTC on ccycloud.quasar-xycaha.root.comops.site
build id 4457881
I20260621 05:14:35.383802 24468 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 05:14:35.384094 24468 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 05:14:35.389987 24476 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:14:35.390458 24474 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:14:35.390530 24473 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:14:35.390465 24468 server_base.cc:1034] running on GCE node
I20260621 05:14:35.391156 24468 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 05:14:35.391618 24468 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 05:14:35.392839 24468 hybrid_clock.cc:648] HybridClock initialized: now 1782018875392824 us; error 29 us; skew 500 ppm
Jun 21 05:14:35 dist-test-slave-vrq1 krb5kdc[24455](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782018875, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 05:14:35.395836 24468 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260621 05:14:35.396309 24468 webserver.cc:460] Webserver started at http://127.0.0.1:34823/ using document root <none> and password file <none>
I20260621 05:14:35.396589 24468 fs_manager.cc:359] Metadata directory not provided
I20260621 05:14:35.396641 24468 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 05:14:35.396780 24468 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 05:14:35.398216 24468 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1782018616648578-3598-0/minicluster-data/master-0/data/instance:
uuid: "86393e4eddd94f8f80dd2a0e72a3f6ed"
format_stamp: "Formatted at 2026-06-21 05:14:35 on dist-test-slave-vrq1"
server_key: "b1638bf7ac44365f599f78f5714cb2a0"
server_key_iv: "48d140c17053faa652c64d27509bbd22"
server_key_version: "encryptionkey@0"
I20260621 05:14:35.398805 24468 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1782018616648578-3598-0/minicluster-data/master-0/wal/instance:
uuid: "86393e4eddd94f8f80dd2a0e72a3f6ed"
format_stamp: "Formatted at 2026-06-21 05:14:35 on dist-test-slave-vrq1"
server_key: "b1638bf7ac44365f599f78f5714cb2a0"
server_key_iv: "48d140c17053faa652c64d27509bbd22"
server_key_version: "encryptionkey@0"
I20260621 05:14:35.400779 24468 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.004s sys 0.000s
I20260621 05:14:35.402012 24483 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 05:14:35.402433 24468 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260621 05:14:35.402544 24468 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1782018616648578-3598-0/minicluster-data/master-0/data,/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1782018616648578-3598-0/minicluster-data/master-0/wal
uuid: "86393e4eddd94f8f80dd2a0e72a3f6ed"
format_stamp: "Formatted at 2026-06-21 05:14:35 on dist-test-slave-vrq1"
server_key: "b1638bf7ac44365f599f78f5714cb2a0"
server_key_iv: "48d140c17053faa652c64d27509bbd22"
server_key_version: "encryptionkey@0"
I20260621 05:14:35.402642 24468 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1782018616648578-3598-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1782018616648578-3598-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1782018616648578-3598-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 05:14:35.411744 24468 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 05:14:35.414129 24468 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 05:14:35.414278 24468 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 05:14:35.418651 24468 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:33163
I20260621 05:14:35.418679 24535 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:33163 every 8 connection(s)
I20260621 05:14:35.419054 24468 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1782018616648578-3598-0/minicluster-data/master-0/data/info.pb
I20260621 05:14:35.420161 24536 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 05:14:35.422531 24536 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 86393e4eddd94f8f80dd2a0e72a3f6ed: Bootstrap starting.
I20260621 05:14:35.422822 3598 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWZOx2_/build/release/bin/kudu as pid 24468
I20260621 05:14:35.422971 3598 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1782018616648578-3598-0/minicluster-data/master-0/wal/instance
I20260621 05:14:35.423162 3598 external_mini_cluster.cc:1428] Setting key 9b49a1dd866e1c7573b552df5b66988a
I20260621 05:14:35.423401 24536 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 86393e4eddd94f8f80dd2a0e72a3f6ed: Neither blocks nor log segments found. Creating new log.
I20260621 05:14:35.423771 24536 log.cc:824] T 00000000000000000000000000000000 P 86393e4eddd94f8f80dd2a0e72a3f6ed: Log is configured to *not* fsync() on all Append() calls
I20260621 05:14:35.424608 24536 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 86393e4eddd94f8f80dd2a0e72a3f6ed: No bootstrap required, opened a new log
I20260621 05:14:35.427106 24536 raft_consensus.cc:348] T 00000000000000000000000000000000 P 86393e4eddd94f8f80dd2a0e72a3f6ed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86393e4eddd94f8f80dd2a0e72a3f6ed" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33163 } }
I20260621 05:14:35.427364 24536 raft_consensus.cc:374] T 00000000000000000000000000000000 P 86393e4eddd94f8f80dd2a0e72a3f6ed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 05:14:35.427423 24536 raft_consensus.cc:729] T 00000000000000000000000000000000 P 86393e4eddd94f8f80dd2a0e72a3f6ed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 86393e4eddd94f8f80dd2a0e72a3f6ed, State: Initialized, Role: FOLLOWER
I20260621 05:14:35.427564 24536 consensus_queue.cc:260] T 00000000000000000000000000000000 P 86393e4eddd94f8f80dd2a0e72a3f6ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86393e4eddd94f8f80dd2a0e72a3f6ed" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33163 } }
I20260621 05:14:35.427675 24536 raft_consensus.cc:388] T 00000000000000000000000000000000 P 86393e4eddd94f8f80dd2a0e72a3f6ed [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260621 05:14:35.427723 24536 raft_consensus.cc:482] T 00000000000000000000000000000000 P 86393e4eddd94f8f80dd2a0e72a3f6ed [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260621 05:14:35.427778 24536 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 86393e4eddd94f8f80dd2a0e72a3f6ed [term 0 FOLLOWER]: Advancing to term 1
I20260621 05:14:35.428574 24536 raft_consensus.cc:504] T 00000000000000000000000000000000 P 86393e4eddd94f8f80dd2a0e72a3f6ed [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86393e4eddd94f8f80dd2a0e72a3f6ed" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33163 } }
I20260621 05:14:35.428692 24536 leader_election.cc:302] T 00000000000000000000000000000000 P 86393e4eddd94f8f80dd2a0e72a3f6ed [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 86393e4eddd94f8f80dd2a0e72a3f6ed; no voters:
I20260621 05:14:35.428889 24536 leader_election.cc:288] T 00000000000000000000000000000000 P 86393e4eddd94f8f80dd2a0e72a3f6ed [CANDIDATE]: Term 1 election: Requested vote from peers
I20260621 05:14:35.429147 24542 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 86393e4eddd94f8f80dd2a0e72a3f6ed [term 1 FOLLOWER]: Leader election won for term 1
I20260621 05:14:35.429292 24536 sys_catalog.cc:564] T 00000000000000000000000000000000 P 86393e4eddd94f8f80dd2a0e72a3f6ed [sys.catalog]: configured and running, proceeding with master startup.
I20260621 05:14:35.429317 24542 raft_consensus.cc:686] T 00000000000000000000000000000000 P 86393e4eddd94f8f80dd2a0e72a3f6ed [term 1 LEADER]: Becoming Leader. State: Replica: 86393e4eddd94f8f80dd2a0e72a3f6ed, State: Running, Role: LEADER
I20260621 05:14:35.429870 24542 consensus_queue.cc:237] T 00000000000000000000000000000000 P 86393e4eddd94f8f80dd2a0e72a3f6ed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86393e4eddd94f8f80dd2a0e72a3f6ed" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33163 } }
I20260621 05:14:35.430334 24542 sys_catalog.cc:455] T 00000000000000000000000000000000 P 86393e4eddd94f8f80dd2a0e72a3f6ed [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "86393e4eddd94f8f80dd2a0e72a3f6ed" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86393e4eddd94f8f80dd2a0e72a3f6ed" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33163 } } }
I20260621 05:14:35.430371 24541 sys_catalog.cc:455] T 00000000000000000000000000000000 P 86393e4eddd94f8f80dd2a0e72a3f6ed [sys.catalog]: SysCatalogTable state changed. Reason: New leader 86393e4eddd94f8f80dd2a0e72a3f6ed. Latest consensus state: current_term: 1 leader_uuid: "86393e4eddd94f8f80dd2a0e72a3f6ed" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86393e4eddd94f8f80dd2a0e72a3f6ed" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33163 } } }
I20260621 05:14:35.430524 24542 sys_catalog.cc:458] T 00000000000000000000000000000000 P 86393e4eddd94f8f80dd2a0e72a3f6ed [sys.catalog]: This master's current role is: LEADER
I20260621 05:14:35.430536 24541 sys_catalog.cc:458] T 00000000000000000000000000000000 P 86393e4eddd94f8f80dd2a0e72a3f6ed [sys.catalog]: This master's current role is: LEADER
W20260621 05:14:35.431527 24556 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 86393e4eddd94f8f80dd2a0e72a3f6ed: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260621 05:14:35.431594 24556 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260621 05:14:35.431752 24554 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260621 05:14:35.431993 24554 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Jun 21 05:14:35 dist-test-slave-vrq1 krb5kdc[24455](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782018875, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260621 05:14:35.436036 24554 catalog_manager.cc:1347] Generated new cluster ID: 024403e75f51400592277eb9d0dd52c8
I20260621 05:14:35.436125 24554 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260621 05:14:35.438563 24539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:14:35.423816 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.16:34575 (local address 127.0.0.1:33163)
0621 05:14:35.424045 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0621 05:14:35.424054 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0621 05:14:35.424089 (+ 35us) server_negotiation.cc:407] Connection header received
0621 05:14:35.424719 (+ 630us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:14:35.424729 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:14:35.424803 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:14:35.424964 (+ 161us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:14:35.425629 (+ 665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:14:35.427120 (+ 1491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:14:35.428127 (+ 1007us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:14:35.428801 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:14:35.428940 (+ 139us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:14:35.434327 (+ 5387us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:14:35.434333 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:14:35.434333 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:14:35.434350 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:14:35.436766 (+ 2416us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:14:35.437311 (+ 545us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:14:35.437314 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:14:35.437315 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:14:35.437422 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:14:35.437685 (+ 263us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:14:35.437686 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:14:35.437687 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:14:35.437973 (+ 286us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:14:35.438178 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0621 05:14:35.438269 (+ 91us) server_negotiation.cc:299] Negotiation successful
0621 05:14:35.438380 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":171,"thread_start_us":111,"threads_started":1}
I20260621 05:14:35.439790 3598 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260621 05:14:35.448083 24539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:14:35.441374 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.16:38899 (local address 127.0.0.1:33163)
0621 05:14:35.441427 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0621 05:14:35.441429 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:14:35.441507 (+ 78us) server_negotiation.cc:407] Connection header received
0621 05:14:35.441574 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:14:35.441575 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:14:35.441587 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:14:35.441643 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:14:35.442196 (+ 553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:14:35.442725 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:14:35.443558 (+ 833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:14:35.443803 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:14:35.443893 (+ 90us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:14:35.445269 (+ 1376us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:14:35.445270 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:14:35.445270 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:14:35.445279 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:14:35.446758 (+ 1479us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:14:35.447230 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:14:35.447231 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:14:35.447232 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:14:35.447293 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:14:35.447545 (+ 252us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:14:35.447546 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:14:35.447547 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:14:35.447672 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:14:35.447787 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0621 05:14:35.447902 (+ 115us) server_negotiation.cc:299] Negotiation successful
0621 05:14:35.447968 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260621 05:14:35.454267 24554 catalog_manager.cc:1370] Generated new certificate authority record
I20260621 05:14:35.455096 24554 catalog_manager.cc:1504] Loading token signing keys...
I20260621 05:14:35.460054 24554 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 86393e4eddd94f8f80dd2a0e72a3f6ed: Generated new TSK 0
I20260621 05:14:35.460289 24554 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260621 05:14:35.463475 3598 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWZOx2_/build/release/bin/kudu with pid 24468
W20260621 05:14:35.467980 24453 connection.cc:535] client connection to 127.0.0.1:33163 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:33163): Connection reset by peer (error 104)
2026-06-21T05:14:35Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (1370 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1782018616648578-3598-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 21 05:14:35 dist-test-slave-vrq1 krb5kdc[24567](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 21 05:14:35 dist-test-slave-vrq1 krb5kdc[24567](info): set up 2 sockets
Jun 21 05:14:35 dist-test-slave-vrq1 krb5kdc[24567](info): commencing operation
krb5kdc: starting...
W20260621 05:14:36.544180 3598 mini_kdc.cc:121] Time spent starting KDC: real 1.062s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1782018616648578-3598-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1782018616648578-3598-0/krb5kdc/test-user.keytab.
Jun 21 05:14:36 dist-test-slave-vrq1 krb5kdc[24567](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782018876, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-21T05:14:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-21T05:14:36Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1782018616648578-3598-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1782018616648578-3598-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260621 05:14:36.680119 3598 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWZOx2_/build/release/bin/kudu
/tmp/dist-test-taskWZOx2_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1782018616648578-3598-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1782018616648578-3598-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1782018616648578-3598-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1782018616648578-3598-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34975
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1782018616648578-3598-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:46471
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:34975
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1782018616648578-3598-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1782018616648578-3598-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1782018616648578-3598-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 05:14:36.762094 24580 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 05:14:36.762372 24580 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 05:14:36.762415 24580 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 05:14:36.763691 24580 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 05:14:36.763769 24580 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 05:14:36.763794 24580 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 05:14:36.763827 24580 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 05:14:36.763856 24580 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 05:14:36.765414 24580 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:46471
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1782018616648578-3598-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1782018616648578-3598-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:34975
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1782018616648578-3598-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34975
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1782018616648578-3598-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24580
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1782018616648578-3598-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 21 Jun 2026 04:16:16 UTC on ccycloud.quasar-xycaha.root.comops.site
build id 4457881
I20260621 05:14:36.765718 24580 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 05:14:36.765956 24580 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 05:14:36.771149 24588 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:14:36.771265 24585 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:14:36.771279 24586 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:14:36.771400 24580 server_base.cc:1034] running on GCE node
I20260621 05:14:36.771683 24580 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 05:14:36.772006 24580 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 05:14:36.773213 24580 hybrid_clock.cc:648] HybridClock initialized: now 1782018876773198 us; error 31 us; skew 500 ppm
Jun 21 05:14:36 dist-test-slave-vrq1 krb5kdc[24567](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782018876, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 05:14:36.775868 24580 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260621 05:14:36.776436 24580 webserver.cc:460] Webserver started at http://127.0.0.1:42583/ using document root <none> and password file <none>
I20260621 05:14:36.776680 24580 fs_manager.cc:359] Metadata directory not provided
I20260621 05:14:36.776751 24580 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 05:14:36.776907 24580 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 05:14:36.777998 24580 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1782018616648578-3598-0/minicluster-data/master-0/data/instance:
uuid: "f1e066c2ee6c4a03ba7ff6aee08e59a6"
format_stamp: "Formatted at 2026-06-21 05:14:36 on dist-test-slave-vrq1"
server_key: "42a2b449f41b5c348957516b7778e475"
server_key_iv: "b4bec81486b7ec1e7326a110628106d8"
server_key_version: "encryptionkey@0"
I20260621 05:14:36.778386 24580 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1782018616648578-3598-0/minicluster-data/master-0/wal/instance:
uuid: "f1e066c2ee6c4a03ba7ff6aee08e59a6"
format_stamp: "Formatted at 2026-06-21 05:14:36 on dist-test-slave-vrq1"
server_key: "42a2b449f41b5c348957516b7778e475"
server_key_iv: "b4bec81486b7ec1e7326a110628106d8"
server_key_version: "encryptionkey@0"
I20260621 05:14:36.779893 24580 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260621 05:14:36.780808 24595 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 05:14:36.781054 24580 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260621 05:14:36.781206 24580 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1782018616648578-3598-0/minicluster-data/master-0/data,/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1782018616648578-3598-0/minicluster-data/master-0/wal
uuid: "f1e066c2ee6c4a03ba7ff6aee08e59a6"
format_stamp: "Formatted at 2026-06-21 05:14:36 on dist-test-slave-vrq1"
server_key: "42a2b449f41b5c348957516b7778e475"
server_key_iv: "b4bec81486b7ec1e7326a110628106d8"
server_key_version: "encryptionkey@0"
I20260621 05:14:36.781299 24580 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1782018616648578-3598-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1782018616648578-3598-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1782018616648578-3598-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 05:14:36.793550 24580 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 05:14:36.795802 24580 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 05:14:36.795969 24580 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 05:14:36.800202 24580 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:34975
I20260621 05:14:36.800259 24647 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:34975 every 8 connection(s)
I20260621 05:14:36.800640 24580 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1782018616648578-3598-0/minicluster-data/master-0/data/info.pb
I20260621 05:14:36.801651 24648 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 05:14:36.804071 24648 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f1e066c2ee6c4a03ba7ff6aee08e59a6: Bootstrap starting.
I20260621 05:14:36.804643 24648 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f1e066c2ee6c4a03ba7ff6aee08e59a6: Neither blocks nor log segments found. Creating new log.
I20260621 05:14:36.804884 24648 log.cc:824] T 00000000000000000000000000000000 P f1e066c2ee6c4a03ba7ff6aee08e59a6: Log is configured to *not* fsync() on all Append() calls
I20260621 05:14:36.805665 24648 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f1e066c2ee6c4a03ba7ff6aee08e59a6: No bootstrap required, opened a new log
I20260621 05:14:36.805740 3598 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWZOx2_/build/release/bin/kudu as pid 24580
I20260621 05:14:36.805856 3598 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1782018616648578-3598-0/minicluster-data/master-0/wal/instance
I20260621 05:14:36.806039 3598 external_mini_cluster.cc:1428] Setting key 68889e63de31761ea37d7b415d52ce5f
I20260621 05:14:36.807798 24648 raft_consensus.cc:348] T 00000000000000000000000000000000 P f1e066c2ee6c4a03ba7ff6aee08e59a6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f1e066c2ee6c4a03ba7ff6aee08e59a6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34975 } }
I20260621 05:14:36.807948 24648 raft_consensus.cc:374] T 00000000000000000000000000000000 P f1e066c2ee6c4a03ba7ff6aee08e59a6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 05:14:36.807971 24648 raft_consensus.cc:729] T 00000000000000000000000000000000 P f1e066c2ee6c4a03ba7ff6aee08e59a6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1e066c2ee6c4a03ba7ff6aee08e59a6, State: Initialized, Role: FOLLOWER
I20260621 05:14:36.808117 24648 consensus_queue.cc:260] T 00000000000000000000000000000000 P f1e066c2ee6c4a03ba7ff6aee08e59a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f1e066c2ee6c4a03ba7ff6aee08e59a6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34975 } }
I20260621 05:14:36.808208 24648 raft_consensus.cc:388] T 00000000000000000000000000000000 P f1e066c2ee6c4a03ba7ff6aee08e59a6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260621 05:14:36.808245 24648 raft_consensus.cc:482] T 00000000000000000000000000000000 P f1e066c2ee6c4a03ba7ff6aee08e59a6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260621 05:14:36.808298 24648 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f1e066c2ee6c4a03ba7ff6aee08e59a6 [term 0 FOLLOWER]: Advancing to term 1
I20260621 05:14:36.808900 24648 raft_consensus.cc:504] T 00000000000000000000000000000000 P f1e066c2ee6c4a03ba7ff6aee08e59a6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f1e066c2ee6c4a03ba7ff6aee08e59a6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34975 } }
I20260621 05:14:36.808995 24648 leader_election.cc:302] T 00000000000000000000000000000000 P f1e066c2ee6c4a03ba7ff6aee08e59a6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f1e066c2ee6c4a03ba7ff6aee08e59a6; no voters:
I20260621 05:14:36.809242 24648 leader_election.cc:288] T 00000000000000000000000000000000 P f1e066c2ee6c4a03ba7ff6aee08e59a6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260621 05:14:36.809269 24653 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f1e066c2ee6c4a03ba7ff6aee08e59a6 [term 1 FOLLOWER]: Leader election won for term 1
I20260621 05:14:36.809381 24653 raft_consensus.cc:686] T 00000000000000000000000000000000 P f1e066c2ee6c4a03ba7ff6aee08e59a6 [term 1 LEADER]: Becoming Leader. State: Replica: f1e066c2ee6c4a03ba7ff6aee08e59a6, State: Running, Role: LEADER
I20260621 05:14:36.809527 24648 sys_catalog.cc:564] T 00000000000000000000000000000000 P f1e066c2ee6c4a03ba7ff6aee08e59a6 [sys.catalog]: configured and running, proceeding with master startup.
I20260621 05:14:36.809533 24653 consensus_queue.cc:237] T 00000000000000000000000000000000 P f1e066c2ee6c4a03ba7ff6aee08e59a6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f1e066c2ee6c4a03ba7ff6aee08e59a6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34975 } }
I20260621 05:14:36.810016 24655 sys_catalog.cc:455] T 00000000000000000000000000000000 P f1e066c2ee6c4a03ba7ff6aee08e59a6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f1e066c2ee6c4a03ba7ff6aee08e59a6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f1e066c2ee6c4a03ba7ff6aee08e59a6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34975 } } }
I20260621 05:14:36.810285 24655 sys_catalog.cc:458] T 00000000000000000000000000000000 P f1e066c2ee6c4a03ba7ff6aee08e59a6 [sys.catalog]: This master's current role is: LEADER
I20260621 05:14:36.810204 24654 sys_catalog.cc:455] T 00000000000000000000000000000000 P f1e066c2ee6c4a03ba7ff6aee08e59a6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f1e066c2ee6c4a03ba7ff6aee08e59a6. Latest consensus state: current_term: 1 leader_uuid: "f1e066c2ee6c4a03ba7ff6aee08e59a6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f1e066c2ee6c4a03ba7ff6aee08e59a6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34975 } } }
I20260621 05:14:36.810559 24654 sys_catalog.cc:458] T 00000000000000000000000000000000 P f1e066c2ee6c4a03ba7ff6aee08e59a6 [sys.catalog]: This master's current role is: LEADER
I20260621 05:14:36.810750 24662 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260621 05:14:36.811317 24662 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260621 05:14:36.813086 24662 catalog_manager.cc:1347] Generated new cluster ID: 7c64bece7b0942c3a38105a0fecd583b
I20260621 05:14:36.813145 24662 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260621 05:14:36.813459 24652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:14:36.806600 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.16:36913 (local address 127.0.0.1:34975)
0621 05:14:36.806935 (+ 335us) server_negotiation.cc:206] Beginning negotiation
0621 05:14:36.806944 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0621 05:14:36.806970 (+ 26us) server_negotiation.cc:407] Connection header received
0621 05:14:36.807796 (+ 826us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:14:36.807805 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:14:36.807880 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:14:36.808095 (+ 215us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:14:36.808418 (+ 323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:14:36.809910 (+ 1492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:14:36.811085 (+ 1175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:14:36.811705 (+ 620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:14:36.811819 (+ 114us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:14:36.812692 (+ 873us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:14:36.812697 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:14:36.812697 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 05:14:36.812927 (+ 230us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:14:36.812966 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 05:14:36.812982 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:14:36.813077 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0621 05:14:36.813219 (+ 142us) server_negotiation.cc:299] Negotiation successful
0621 05:14:36.813250 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":275,"thread_start_us":229,"threads_started":1}
I20260621 05:14:36.814533 3598 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260621 05:14:36.825814 24662 catalog_manager.cc:1370] Generated new certificate authority record
I20260621 05:14:36.826622 24662 catalog_manager.cc:1504] Loading token signing keys...
I20260621 05:14:36.829195 24652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:14:36.825956 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.16:33577 (local address 127.0.0.1:34975)
0621 05:14:36.826004 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0621 05:14:36.826007 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0621 05:14:36.826209 (+ 202us) server_negotiation.cc:407] Connection header received
0621 05:14:36.826293 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:14:36.826295 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:14:36.826307 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:14:36.826353 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:14:36.826525 (+ 172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:14:36.827101 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:14:36.827882 (+ 781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:14:36.828129 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:14:36.828209 (+ 80us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:14:36.828931 (+ 722us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:14:36.828932 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:14:36.828933 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 05:14:36.828938 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:14:36.828953 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 05:14:36.828958 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:14:36.829012 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0621 05:14:36.829071 (+ 59us) server_negotiation.cc:299] Negotiation successful
0621 05:14:36.829085 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260621 05:14:36.839839 24662 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f1e066c2ee6c4a03ba7ff6aee08e59a6: Generated new TSK 0
I20260621 05:14:36.840091 24662 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260621 05:14:36.841919 3598 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWZOx2_/build/release/bin/kudu with pid 24580
W20260621 05:14:36.846313 24565 connection.cc:535] client connection to 127.0.0.1:34975 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:34975): Connection reset by peer (error 104)
2026-06-21T05:14:36Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1378 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1782018616648578-3598-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 21 05:14:36 dist-test-slave-vrq1 krb5kdc[24679](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 21 05:14:36 dist-test-slave-vrq1 krb5kdc[24679](info): set up 2 sockets
Jun 21 05:14:36 dist-test-slave-vrq1 krb5kdc[24679](info): commencing operation
krb5kdc: starting...
W20260621 05:14:37.887535 3598 mini_kdc.cc:121] Time spent starting KDC: real 1.027s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1782018616648578-3598-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1782018616648578-3598-0/krb5kdc/test-user.keytab.
Jun 21 05:14:37 dist-test-slave-vrq1 krb5kdc[24679](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782018877, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-21T05:14:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-21T05:14:37Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1782018616648578-3598-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1782018616648578-3598-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260621 05:14:38.009758 3598 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWZOx2_/build/release/bin/kudu
/tmp/dist-test-taskWZOx2_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1782018616648578-3598-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1782018616648578-3598-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1782018616648578-3598-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1782018616648578-3598-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45325
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1782018616648578-3598-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42331
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45325
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1782018616648578-3598-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1782018616648578-3598-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1782018616648578-3598-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 05:14:38.092208 24692 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 05:14:38.092487 24692 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 05:14:38.092530 24692 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 05:14:38.093889 24692 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 05:14:38.093972 24692 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 05:14:38.094000 24692 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 05:14:38.094031 24692 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 05:14:38.094055 24692 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 05:14:38.095793 24692 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42331
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1782018616648578-3598-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1782018616648578-3598-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45325
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1782018616648578-3598-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45325
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1782018616648578-3598-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24692
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1782018616648578-3598-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 21 Jun 2026 04:16:16 UTC on ccycloud.quasar-xycaha.root.comops.site
build id 4457881
I20260621 05:14:38.096097 24692 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 05:14:38.096333 24692 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 05:14:38.101464 24698 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:14:38.101480 24697 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:14:38.101502 24692 server_base.cc:1034] running on GCE node
W20260621 05:14:38.101466 24700 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:14:38.102051 24692 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 05:14:38.102394 24692 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 05:14:38.103616 24692 hybrid_clock.cc:648] HybridClock initialized: now 1782018878103601 us; error 30 us; skew 500 ppm
Jun 21 05:14:38 dist-test-slave-vrq1 krb5kdc[24679](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782018878, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 05:14:38.112032 24692 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260621 05:14:38.112630 24692 webserver.cc:460] Webserver started at http://127.0.0.1:37291/ using document root <none> and password file <none>
I20260621 05:14:38.112919 24692 fs_manager.cc:359] Metadata directory not provided
I20260621 05:14:38.112999 24692 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 05:14:38.113188 24692 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 05:14:38.114404 24692 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1782018616648578-3598-0/minicluster-data/master-0/data/instance:
uuid: "ac370f10eccf4e36ab223fbd8ccbf5cf"
format_stamp: "Formatted at 2026-06-21 05:14:38 on dist-test-slave-vrq1"
server_key: "a72162032baa17c36376634e24eca945"
server_key_iv: "6449beaaa9df65a615affd2a6bb719a4"
server_key_version: "encryptionkey@0"
I20260621 05:14:38.114813 24692 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1782018616648578-3598-0/minicluster-data/master-0/wal/instance:
uuid: "ac370f10eccf4e36ab223fbd8ccbf5cf"
format_stamp: "Formatted at 2026-06-21 05:14:38 on dist-test-slave-vrq1"
server_key: "a72162032baa17c36376634e24eca945"
server_key_iv: "6449beaaa9df65a615affd2a6bb719a4"
server_key_version: "encryptionkey@0"
I20260621 05:14:38.116499 24692 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.004s
I20260621 05:14:38.117533 24707 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 05:14:38.117810 24692 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260621 05:14:38.117935 24692 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1782018616648578-3598-0/minicluster-data/master-0/data,/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1782018616648578-3598-0/minicluster-data/master-0/wal
uuid: "ac370f10eccf4e36ab223fbd8ccbf5cf"
format_stamp: "Formatted at 2026-06-21 05:14:38 on dist-test-slave-vrq1"
server_key: "a72162032baa17c36376634e24eca945"
server_key_iv: "6449beaaa9df65a615affd2a6bb719a4"
server_key_version: "encryptionkey@0"
I20260621 05:14:38.118038 24692 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1782018616648578-3598-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1782018616648578-3598-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1782018616648578-3598-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 05:14:38.121941 24692 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 05:14:38.124610 24692 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 05:14:38.124787 24692 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 05:14:38.129256 24692 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45325
I20260621 05:14:38.129287 24759 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45325 every 8 connection(s)
I20260621 05:14:38.129664 24692 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1782018616648578-3598-0/minicluster-data/master-0/data/info.pb
I20260621 05:14:38.130697 24760 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 05:14:38.133028 24760 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ac370f10eccf4e36ab223fbd8ccbf5cf: Bootstrap starting.
I20260621 05:14:38.133571 24760 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ac370f10eccf4e36ab223fbd8ccbf5cf: Neither blocks nor log segments found. Creating new log.
I20260621 05:14:38.133797 24760 log.cc:824] T 00000000000000000000000000000000 P ac370f10eccf4e36ab223fbd8ccbf5cf: Log is configured to *not* fsync() on all Append() calls
I20260621 05:14:38.134470 24760 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ac370f10eccf4e36ab223fbd8ccbf5cf: No bootstrap required, opened a new log
I20260621 05:14:38.135290 3598 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWZOx2_/build/release/bin/kudu as pid 24692
I20260621 05:14:38.135424 3598 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1782018616648578-3598-0/minicluster-data/master-0/wal/instance
I20260621 05:14:38.135596 3598 external_mini_cluster.cc:1428] Setting key 8d0b482901803de9495c49640ec6836f
I20260621 05:14:38.136524 24760 raft_consensus.cc:348] T 00000000000000000000000000000000 P ac370f10eccf4e36ab223fbd8ccbf5cf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac370f10eccf4e36ab223fbd8ccbf5cf" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45325 } }
I20260621 05:14:38.136718 24760 raft_consensus.cc:374] T 00000000000000000000000000000000 P ac370f10eccf4e36ab223fbd8ccbf5cf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 05:14:38.136762 24760 raft_consensus.cc:729] T 00000000000000000000000000000000 P ac370f10eccf4e36ab223fbd8ccbf5cf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ac370f10eccf4e36ab223fbd8ccbf5cf, State: Initialized, Role: FOLLOWER
I20260621 05:14:38.136929 24760 consensus_queue.cc:260] T 00000000000000000000000000000000 P ac370f10eccf4e36ab223fbd8ccbf5cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac370f10eccf4e36ab223fbd8ccbf5cf" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45325 } }
I20260621 05:14:38.137020 24760 raft_consensus.cc:388] T 00000000000000000000000000000000 P ac370f10eccf4e36ab223fbd8ccbf5cf [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260621 05:14:38.137050 24760 raft_consensus.cc:482] T 00000000000000000000000000000000 P ac370f10eccf4e36ab223fbd8ccbf5cf [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260621 05:14:38.137118 24760 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ac370f10eccf4e36ab223fbd8ccbf5cf [term 0 FOLLOWER]: Advancing to term 1
I20260621 05:14:38.137802 24764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:14:38.136165 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.16:34987 (local address 127.0.0.1:45325)
0621 05:14:38.136350 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0621 05:14:38.136358 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0621 05:14:38.136382 (+ 24us) server_negotiation.cc:407] Connection header received
0621 05:14:38.136480 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:14:38.136485 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:14:38.136509 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:14:38.136651 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:14:38.137346 (+ 695us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:14:38.137349 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:14:38.137349 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 05:14:38.137352 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:14:38.137391 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 05:14:38.137402 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:14:38.137454 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0621 05:14:38.137542 (+ 88us) server_negotiation.cc:299] Negotiation successful
0621 05:14:38.137577 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":129,"thread_start_us":100,"threads_started":1}
I20260621 05:14:38.138203 24760 raft_consensus.cc:504] T 00000000000000000000000000000000 P ac370f10eccf4e36ab223fbd8ccbf5cf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac370f10eccf4e36ab223fbd8ccbf5cf" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45325 } }
I20260621 05:14:38.138319 24760 leader_election.cc:302] T 00000000000000000000000000000000 P ac370f10eccf4e36ab223fbd8ccbf5cf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ac370f10eccf4e36ab223fbd8ccbf5cf; no voters:
I20260621 05:14:38.138504 24760 leader_election.cc:288] T 00000000000000000000000000000000 P ac370f10eccf4e36ab223fbd8ccbf5cf [CANDIDATE]: Term 1 election: Requested vote from peers
I20260621 05:14:38.138515 24765 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ac370f10eccf4e36ab223fbd8ccbf5cf [term 1 FOLLOWER]: Leader election won for term 1
I20260621 05:14:38.138669 24765 raft_consensus.cc:686] T 00000000000000000000000000000000 P ac370f10eccf4e36ab223fbd8ccbf5cf [term 1 LEADER]: Becoming Leader. State: Replica: ac370f10eccf4e36ab223fbd8ccbf5cf, State: Running, Role: LEADER
I20260621 05:14:38.138695 3598 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260621 05:14:38.138767 24760 sys_catalog.cc:564] T 00000000000000000000000000000000 P ac370f10eccf4e36ab223fbd8ccbf5cf [sys.catalog]: configured and running, proceeding with master startup.
I20260621 05:14:38.138795 24765 consensus_queue.cc:237] T 00000000000000000000000000000000 P ac370f10eccf4e36ab223fbd8ccbf5cf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac370f10eccf4e36ab223fbd8ccbf5cf" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45325 } }
I20260621 05:14:38.139091 24765 sys_catalog.cc:455] T 00000000000000000000000000000000 P ac370f10eccf4e36ab223fbd8ccbf5cf [sys.catalog]: SysCatalogTable state changed. Reason: New leader ac370f10eccf4e36ab223fbd8ccbf5cf. Latest consensus state: current_term: 1 leader_uuid: "ac370f10eccf4e36ab223fbd8ccbf5cf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac370f10eccf4e36ab223fbd8ccbf5cf" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45325 } } }
I20260621 05:14:38.139102 24767 sys_catalog.cc:455] T 00000000000000000000000000000000 P ac370f10eccf4e36ab223fbd8ccbf5cf [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ac370f10eccf4e36ab223fbd8ccbf5cf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac370f10eccf4e36ab223fbd8ccbf5cf" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45325 } } }
I20260621 05:14:38.139204 24765 sys_catalog.cc:458] T 00000000000000000000000000000000 P ac370f10eccf4e36ab223fbd8ccbf5cf [sys.catalog]: This master's current role is: LEADER
I20260621 05:14:38.139226 24767 sys_catalog.cc:458] T 00000000000000000000000000000000 P ac370f10eccf4e36ab223fbd8ccbf5cf [sys.catalog]: This master's current role is: LEADER
I20260621 05:14:38.139928 24771 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260621 05:14:38.140455 24771 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260621 05:14:38.141566 24764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:14:38.140280 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.16:36721 (local address 127.0.0.1:45325)
0621 05:14:38.140314 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0621 05:14:38.140316 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:14:38.140562 (+ 246us) server_negotiation.cc:407] Connection header received
0621 05:14:38.140664 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:14:38.140665 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:14:38.140671 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:14:38.140699 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:14:38.141362 (+ 663us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:14:38.141363 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:14:38.141363 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 05:14:38.141367 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:14:38.141383 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 05:14:38.141387 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:14:38.141454 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0621 05:14:38.141499 (+ 45us) server_negotiation.cc:299] Negotiation successful
0621 05:14:38.141512 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260621 05:14:38.142254 24771 catalog_manager.cc:1347] Generated new cluster ID: e5e271f2b0c34bb6a799906483cf14c9
I20260621 05:14:38.142304 24771 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260621 05:14:38.153726 24771 catalog_manager.cc:1370] Generated new certificate authority record
I20260621 05:14:38.153931 24771 catalog_manager.cc:1504] Loading token signing keys...
I20260621 05:14:38.162492 24771 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ac370f10eccf4e36ab223fbd8ccbf5cf: Generated new TSK 0
I20260621 05:14:38.162783 24771 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260621 05:14:38.170289 3598 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWZOx2_/build/release/bin/kudu with pid 24692
2026-06-21T05:14:38Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1328 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1782018616648578-3598-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 21 05:14:38 dist-test-slave-vrq1 krb5kdc[24791](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 21 05:14:38 dist-test-slave-vrq1 krb5kdc[24791](info): set up 2 sockets
Jun 21 05:14:38 dist-test-slave-vrq1 krb5kdc[24791](info): commencing operation
krb5kdc: starting...
W20260621 05:14:39.224014 3598 mini_kdc.cc:121] Time spent starting KDC: real 1.035s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1782018616648578-3598-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1782018616648578-3598-0/krb5kdc/test-user.keytab.
Jun 21 05:14:39 dist-test-slave-vrq1 krb5kdc[24791](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782018879, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-21T05:14:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-21T05:14:39Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1782018616648578-3598-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1782018616648578-3598-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260621 05:14:39.349716 3598 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWZOx2_/build/release/bin/kudu
/tmp/dist-test-taskWZOx2_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1782018616648578-3598-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1782018616648578-3598-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1782018616648578-3598-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1782018616648578-3598-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40023
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1782018616648578-3598-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42185
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40023
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1782018616648578-3598-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1782018616648578-3598-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1782018616648578-3598-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 05:14:39.432883 24804 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 05:14:39.433138 24804 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 05:14:39.433182 24804 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 05:14:39.434337 24804 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 05:14:39.434412 24804 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 05:14:39.434449 24804 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 05:14:39.434506 24804 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 05:14:39.434533 24804 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 05:14:39.436144 24804 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42185
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1782018616648578-3598-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1782018616648578-3598-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40023
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1782018616648578-3598-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40023
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1782018616648578-3598-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24804
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1782018616648578-3598-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 21 Jun 2026 04:16:16 UTC on ccycloud.quasar-xycaha.root.comops.site
build id 4457881
I20260621 05:14:39.436429 24804 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 05:14:39.436681 24804 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 05:14:39.441847 24812 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:14:39.441998 24804 server_base.cc:1034] running on GCE node
W20260621 05:14:39.441892 24809 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:14:39.441890 24810 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:14:39.442417 24804 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 05:14:39.442777 24804 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 05:14:39.443945 24804 hybrid_clock.cc:648] HybridClock initialized: now 1782018879443934 us; error 47 us; skew 500 ppm
Jun 21 05:14:39 dist-test-slave-vrq1 krb5kdc[24791](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782018879, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 05:14:39.447847 24804 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260621 05:14:39.448333 24804 webserver.cc:460] Webserver started at http://127.0.0.1:45025/ using document root <none> and password file <none>
I20260621 05:14:39.448540 24804 fs_manager.cc:359] Metadata directory not provided
I20260621 05:14:39.448587 24804 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 05:14:39.448680 24804 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 05:14:39.449715 24804 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1782018616648578-3598-0/minicluster-data/master-0/data/instance:
uuid: "23857770521941a2a512e6c684f6bbd6"
format_stamp: "Formatted at 2026-06-21 05:14:39 on dist-test-slave-vrq1"
server_key: "9e7c3b34378273129a1319bd4cce3f18"
server_key_iv: "b67d5150895e9a7472c8cf0e9ff89449"
server_key_version: "encryptionkey@0"
I20260621 05:14:39.450027 24804 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1782018616648578-3598-0/minicluster-data/master-0/wal/instance:
uuid: "23857770521941a2a512e6c684f6bbd6"
format_stamp: "Formatted at 2026-06-21 05:14:39 on dist-test-slave-vrq1"
server_key: "9e7c3b34378273129a1319bd4cce3f18"
server_key_iv: "b67d5150895e9a7472c8cf0e9ff89449"
server_key_version: "encryptionkey@0"
I20260621 05:14:39.451457 24804 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260621 05:14:39.452258 24819 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 05:14:39.452627 24804 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260621 05:14:39.452715 24804 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1782018616648578-3598-0/minicluster-data/master-0/data,/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1782018616648578-3598-0/minicluster-data/master-0/wal
uuid: "23857770521941a2a512e6c684f6bbd6"
format_stamp: "Formatted at 2026-06-21 05:14:39 on dist-test-slave-vrq1"
server_key: "9e7c3b34378273129a1319bd4cce3f18"
server_key_iv: "b67d5150895e9a7472c8cf0e9ff89449"
server_key_version: "encryptionkey@0"
I20260621 05:14:39.452809 24804 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1782018616648578-3598-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1782018616648578-3598-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1782018616648578-3598-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 05:14:39.475857 24804 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 05:14:39.478309 24804 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 05:14:39.478456 24804 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 05:14:39.482970 24804 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40023
I20260621 05:14:39.483007 24871 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40023 every 8 connection(s)
I20260621 05:14:39.483418 24804 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1782018616648578-3598-0/minicluster-data/master-0/data/info.pb
I20260621 05:14:39.484458 24872 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 05:14:39.485154 3598 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWZOx2_/build/release/bin/kudu as pid 24804
I20260621 05:14:39.485275 3598 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1782018616648578-3598-0/minicluster-data/master-0/wal/instance
I20260621 05:14:39.485436 3598 external_mini_cluster.cc:1428] Setting key b456111e1da85938b039339766e41532
I20260621 05:14:39.487679 24872 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 23857770521941a2a512e6c684f6bbd6: Bootstrap starting.
I20260621 05:14:39.488320 24872 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 23857770521941a2a512e6c684f6bbd6: Neither blocks nor log segments found. Creating new log.
I20260621 05:14:39.488627 24872 log.cc:824] T 00000000000000000000000000000000 P 23857770521941a2a512e6c684f6bbd6: Log is configured to *not* fsync() on all Append() calls
I20260621 05:14:39.489416 24872 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 23857770521941a2a512e6c684f6bbd6: No bootstrap required, opened a new log
I20260621 05:14:39.491864 24872 raft_consensus.cc:348] T 00000000000000000000000000000000 P 23857770521941a2a512e6c684f6bbd6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "23857770521941a2a512e6c684f6bbd6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40023 } }
I20260621 05:14:39.492009 24872 raft_consensus.cc:374] T 00000000000000000000000000000000 P 23857770521941a2a512e6c684f6bbd6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 05:14:39.492033 24872 raft_consensus.cc:729] T 00000000000000000000000000000000 P 23857770521941a2a512e6c684f6bbd6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23857770521941a2a512e6c684f6bbd6, State: Initialized, Role: FOLLOWER
I20260621 05:14:39.492179 24872 consensus_queue.cc:260] T 00000000000000000000000000000000 P 23857770521941a2a512e6c684f6bbd6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "23857770521941a2a512e6c684f6bbd6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40023 } }
I20260621 05:14:39.492286 24872 raft_consensus.cc:388] T 00000000000000000000000000000000 P 23857770521941a2a512e6c684f6bbd6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260621 05:14:39.492326 24872 raft_consensus.cc:482] T 00000000000000000000000000000000 P 23857770521941a2a512e6c684f6bbd6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260621 05:14:39.492372 24872 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 23857770521941a2a512e6c684f6bbd6 [term 0 FOLLOWER]: Advancing to term 1
Jun 21 05:14:39 dist-test-slave-vrq1 krb5kdc[24791](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782018879, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260621 05:14:39.493124 24872 raft_consensus.cc:504] T 00000000000000000000000000000000 P 23857770521941a2a512e6c684f6bbd6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "23857770521941a2a512e6c684f6bbd6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40023 } }
I20260621 05:14:39.493252 24872 leader_election.cc:302] T 00000000000000000000000000000000 P 23857770521941a2a512e6c684f6bbd6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 23857770521941a2a512e6c684f6bbd6; no voters:
I20260621 05:14:39.493468 24872 leader_election.cc:288] T 00000000000000000000000000000000 P 23857770521941a2a512e6c684f6bbd6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260621 05:14:39.493546 24877 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 23857770521941a2a512e6c684f6bbd6 [term 1 FOLLOWER]: Leader election won for term 1
I20260621 05:14:39.493748 24877 raft_consensus.cc:686] T 00000000000000000000000000000000 P 23857770521941a2a512e6c684f6bbd6 [term 1 LEADER]: Becoming Leader. State: Replica: 23857770521941a2a512e6c684f6bbd6, State: Running, Role: LEADER
I20260621 05:14:39.493770 24872 sys_catalog.cc:564] T 00000000000000000000000000000000 P 23857770521941a2a512e6c684f6bbd6 [sys.catalog]: configured and running, proceeding with master startup.
I20260621 05:14:39.493952 24877 consensus_queue.cc:237] T 00000000000000000000000000000000 P 23857770521941a2a512e6c684f6bbd6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "23857770521941a2a512e6c684f6bbd6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40023 } }
I20260621 05:14:39.494930 24878 sys_catalog.cc:455] T 00000000000000000000000000000000 P 23857770521941a2a512e6c684f6bbd6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "23857770521941a2a512e6c684f6bbd6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "23857770521941a2a512e6c684f6bbd6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40023 } } }
I20260621 05:14:39.495013 24878 sys_catalog.cc:458] T 00000000000000000000000000000000 P 23857770521941a2a512e6c684f6bbd6 [sys.catalog]: This master's current role is: LEADER
I20260621 05:14:39.495357 24887 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260621 05:14:39.495610 24879 sys_catalog.cc:455] T 00000000000000000000000000000000 P 23857770521941a2a512e6c684f6bbd6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 23857770521941a2a512e6c684f6bbd6. Latest consensus state: current_term: 1 leader_uuid: "23857770521941a2a512e6c684f6bbd6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "23857770521941a2a512e6c684f6bbd6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40023 } } }
I20260621 05:14:39.495807 24879 sys_catalog.cc:458] T 00000000000000000000000000000000 P 23857770521941a2a512e6c684f6bbd6 [sys.catalog]: This master's current role is: LEADER
I20260621 05:14:39.495976 24887 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260621 05:14:39.497848 24875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:14:39.486047 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.16:40069 (local address 127.0.0.1:40023)
0621 05:14:39.486221 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0621 05:14:39.486230 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0621 05:14:39.486263 (+ 33us) server_negotiation.cc:407] Connection header received
0621 05:14:39.486674 (+ 411us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:14:39.486681 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:14:39.486731 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:14:39.486851 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:14:39.487720 (+ 869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:14:39.488964 (+ 1244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:14:39.489790 (+ 826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:14:39.490413 (+ 623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:14:39.490548 (+ 135us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:14:39.493482 (+ 2934us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:14:39.493487 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:14:39.493488 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:14:39.493501 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:14:39.496064 (+ 2563us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:14:39.496573 (+ 509us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:14:39.496576 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:14:39.496576 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:14:39.496697 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:14:39.496940 (+ 243us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:14:39.496940 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:14:39.496941 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:14:39.497265 (+ 324us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:14:39.497442 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0621 05:14:39.497534 (+ 92us) server_negotiation.cc:299] Negotiation successful
0621 05:14:39.497646 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":126,"thread_start_us":89,"threads_started":1}
I20260621 05:14:39.499073 3598 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260621 05:14:39.500530 24887 catalog_manager.cc:1347] Generated new cluster ID: 4685e36253d245fc8b3f3cdf9d450733
I20260621 05:14:39.500588 24887 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260621 05:14:39.509738 24875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:14:39.500735 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.16:37789 (local address 127.0.0.1:40023)
0621 05:14:39.503324 (+ 2589us) server_negotiation.cc:206] Beginning negotiation
0621 05:14:39.503326 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:14:39.503337 (+ 11us) server_negotiation.cc:407] Connection header received
0621 05:14:39.503371 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:14:39.503372 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:14:39.503381 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:14:39.503459 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:14:39.504081 (+ 622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:14:39.504584 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:14:39.505320 (+ 736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:14:39.505584 (+ 264us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:14:39.505735 (+ 151us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:14:39.506988 (+ 1253us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:14:39.506990 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:14:39.506990 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:14:39.506998 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:14:39.508524 (+ 1526us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:14:39.508901 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:14:39.508902 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:14:39.508902 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:14:39.508962 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:14:39.509195 (+ 233us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:14:39.509196 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:14:39.509196 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:14:39.509314 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:14:39.509426 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0621 05:14:39.509554 (+ 128us) server_negotiation.cc:299] Negotiation successful
0621 05:14:39.509618 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2564}
I20260621 05:14:39.520232 24887 catalog_manager.cc:1370] Generated new certificate authority record
I20260621 05:14:39.521050 24887 catalog_manager.cc:1504] Loading token signing keys...
I20260621 05:14:39.528544 24887 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 23857770521941a2a512e6c684f6bbd6: Generated new TSK 0
I20260621 05:14:39.528829 24887 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260621 05:14:39.533833 3598 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWZOx2_/build/release/bin/kudu with pid 24804
W20260621 05:14:39.538244 24789 connection.cc:535] client connection to 127.0.0.1:40023 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:40023): Connection reset by peer (error 104)
2026-06-21T05:14:39Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (1363 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1782018616648578-3598-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 21 05:14:39 dist-test-slave-vrq1 krb5kdc[24903](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 21 05:14:39 dist-test-slave-vrq1 krb5kdc[24903](info): set up 2 sockets
Jun 21 05:14:39 dist-test-slave-vrq1 krb5kdc[24903](info): commencing operation
krb5kdc: starting...
W20260621 05:14:40.591861 3598 mini_kdc.cc:121] Time spent starting KDC: real 1.040s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1782018616648578-3598-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1782018616648578-3598-0/krb5kdc/test-user.keytab.
Jun 21 05:14:40 dist-test-slave-vrq1 krb5kdc[24903](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782018880, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-21T05:14:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-21T05:14:40Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1782018616648578-3598-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1782018616648578-3598-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260621 05:14:40.726931 3598 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWZOx2_/build/release/bin/kudu
/tmp/dist-test-taskWZOx2_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1782018616648578-3598-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1782018616648578-3598-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1782018616648578-3598-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1782018616648578-3598-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:32903
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1782018616648578-3598-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34403
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:32903
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1782018616648578-3598-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1782018616648578-3598-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1782018616648578-3598-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 05:14:40.811201 24916 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 05:14:40.811452 24916 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 05:14:40.811492 24916 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 05:14:40.812698 24916 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 05:14:40.812775 24916 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 05:14:40.812803 24916 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 05:14:40.812834 24916 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 05:14:40.812857 24916 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 05:14:40.814420 24916 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34403
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1782018616648578-3598-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1782018616648578-3598-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:32903
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1782018616648578-3598-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:32903
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1782018616648578-3598-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24916
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1782018616648578-3598-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 21 Jun 2026 04:16:16 UTC on ccycloud.quasar-xycaha.root.comops.site
build id 4457881
I20260621 05:14:40.814720 24916 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 05:14:40.814951 24916 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 05:14:40.820173 24922 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:14:40.820144 24924 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:14:40.820144 24921 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:14:40.820474 24916 server_base.cc:1034] running on GCE node
I20260621 05:14:40.820806 24916 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 05:14:40.821171 24916 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 05:14:40.822381 24916 hybrid_clock.cc:648] HybridClock initialized: now 1782018880822369 us; error 34 us; skew 500 ppm
Jun 21 05:14:40 dist-test-slave-vrq1 krb5kdc[24903](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782018880, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 05:14:40.825594 24916 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260621 05:14:40.826150 24916 webserver.cc:460] Webserver started at http://127.0.0.1:40807/ using document root <none> and password file <none>
I20260621 05:14:40.826396 24916 fs_manager.cc:359] Metadata directory not provided
I20260621 05:14:40.826472 24916 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 05:14:40.826632 24916 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 05:14:40.827863 24916 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1782018616648578-3598-0/minicluster-data/master-0/data/instance:
uuid: "c06fa43548cb4e63b2426a71b5b33741"
format_stamp: "Formatted at 2026-06-21 05:14:40 on dist-test-slave-vrq1"
server_key: "1b6a272d968b0f5082f2a1da8f4fcd43"
server_key_iv: "08ee0230a8638ce820faea75ff83bfbc"
server_key_version: "encryptionkey@0"
I20260621 05:14:40.828285 24916 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1782018616648578-3598-0/minicluster-data/master-0/wal/instance:
uuid: "c06fa43548cb4e63b2426a71b5b33741"
format_stamp: "Formatted at 2026-06-21 05:14:40 on dist-test-slave-vrq1"
server_key: "1b6a272d968b0f5082f2a1da8f4fcd43"
server_key_iv: "08ee0230a8638ce820faea75ff83bfbc"
server_key_version: "encryptionkey@0"
I20260621 05:14:40.829818 24916 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260621 05:14:40.830770 24931 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 05:14:40.831162 24916 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260621 05:14:40.831364 24916 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1782018616648578-3598-0/minicluster-data/master-0/data,/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1782018616648578-3598-0/minicluster-data/master-0/wal
uuid: "c06fa43548cb4e63b2426a71b5b33741"
format_stamp: "Formatted at 2026-06-21 05:14:40 on dist-test-slave-vrq1"
server_key: "1b6a272d968b0f5082f2a1da8f4fcd43"
server_key_iv: "08ee0230a8638ce820faea75ff83bfbc"
server_key_version: "encryptionkey@0"
I20260621 05:14:40.831452 24916 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1782018616648578-3598-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1782018616648578-3598-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1782018616648578-3598-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 05:14:40.852095 24916 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 05:14:40.854348 24916 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 05:14:40.854518 24916 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 05:14:40.858490 24916 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:32903
I20260621 05:14:40.858579 24983 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:32903 every 8 connection(s)
I20260621 05:14:40.858928 24916 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1782018616648578-3598-0/minicluster-data/master-0/data/info.pb
I20260621 05:14:40.859843 24984 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 05:14:40.862093 24984 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c06fa43548cb4e63b2426a71b5b33741: Bootstrap starting.
I20260621 05:14:40.862586 3598 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWZOx2_/build/release/bin/kudu as pid 24916
I20260621 05:14:40.862703 3598 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1782018616648578-3598-0/minicluster-data/master-0/wal/instance
I20260621 05:14:40.862836 24984 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c06fa43548cb4e63b2426a71b5b33741: Neither blocks nor log segments found. Creating new log.
I20260621 05:14:40.862890 3598 external_mini_cluster.cc:1428] Setting key 31400d07bca1257aa8d88bf0a565e769
I20260621 05:14:40.863154 24984 log.cc:824] T 00000000000000000000000000000000 P c06fa43548cb4e63b2426a71b5b33741: Log is configured to *not* fsync() on all Append() calls
I20260621 05:14:40.863968 24984 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c06fa43548cb4e63b2426a71b5b33741: No bootstrap required, opened a new log
I20260621 05:14:40.866266 24984 raft_consensus.cc:348] T 00000000000000000000000000000000 P c06fa43548cb4e63b2426a71b5b33741 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c06fa43548cb4e63b2426a71b5b33741" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32903 } }
I20260621 05:14:40.866444 24984 raft_consensus.cc:374] T 00000000000000000000000000000000 P c06fa43548cb4e63b2426a71b5b33741 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 05:14:40.866477 24984 raft_consensus.cc:729] T 00000000000000000000000000000000 P c06fa43548cb4e63b2426a71b5b33741 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c06fa43548cb4e63b2426a71b5b33741, State: Initialized, Role: FOLLOWER
I20260621 05:14:40.866602 24984 consensus_queue.cc:260] T 00000000000000000000000000000000 P c06fa43548cb4e63b2426a71b5b33741 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c06fa43548cb4e63b2426a71b5b33741" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32903 } }
I20260621 05:14:40.866674 24984 raft_consensus.cc:388] T 00000000000000000000000000000000 P c06fa43548cb4e63b2426a71b5b33741 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260621 05:14:40.866708 24984 raft_consensus.cc:482] T 00000000000000000000000000000000 P c06fa43548cb4e63b2426a71b5b33741 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260621 05:14:40.866755 24984 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c06fa43548cb4e63b2426a71b5b33741 [term 0 FOLLOWER]: Advancing to term 1
I20260621 05:14:40.867417 24984 raft_consensus.cc:504] T 00000000000000000000000000000000 P c06fa43548cb4e63b2426a71b5b33741 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c06fa43548cb4e63b2426a71b5b33741" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32903 } }
I20260621 05:14:40.867524 24984 leader_election.cc:302] T 00000000000000000000000000000000 P c06fa43548cb4e63b2426a71b5b33741 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c06fa43548cb4e63b2426a71b5b33741; no voters:
I20260621 05:14:40.867698 24984 leader_election.cc:288] T 00000000000000000000000000000000 P c06fa43548cb4e63b2426a71b5b33741 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260621 05:14:40.867765 24989 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c06fa43548cb4e63b2426a71b5b33741 [term 1 FOLLOWER]: Leader election won for term 1
I20260621 05:14:40.867954 24984 sys_catalog.cc:564] T 00000000000000000000000000000000 P c06fa43548cb4e63b2426a71b5b33741 [sys.catalog]: configured and running, proceeding with master startup.
I20260621 05:14:40.867967 24989 raft_consensus.cc:686] T 00000000000000000000000000000000 P c06fa43548cb4e63b2426a71b5b33741 [term 1 LEADER]: Becoming Leader. State: Replica: c06fa43548cb4e63b2426a71b5b33741, State: Running, Role: LEADER
I20260621 05:14:40.868120 24989 consensus_queue.cc:237] T 00000000000000000000000000000000 P c06fa43548cb4e63b2426a71b5b33741 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c06fa43548cb4e63b2426a71b5b33741" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32903 } }
I20260621 05:14:40.868866 24991 sys_catalog.cc:455] T 00000000000000000000000000000000 P c06fa43548cb4e63b2426a71b5b33741 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c06fa43548cb4e63b2426a71b5b33741. Latest consensus state: current_term: 1 leader_uuid: "c06fa43548cb4e63b2426a71b5b33741" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c06fa43548cb4e63b2426a71b5b33741" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32903 } } }
I20260621 05:14:40.868943 24991 sys_catalog.cc:458] T 00000000000000000000000000000000 P c06fa43548cb4e63b2426a71b5b33741 [sys.catalog]: This master's current role is: LEADER
I20260621 05:14:40.869179 24990 sys_catalog.cc:455] T 00000000000000000000000000000000 P c06fa43548cb4e63b2426a71b5b33741 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c06fa43548cb4e63b2426a71b5b33741" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c06fa43548cb4e63b2426a71b5b33741" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32903 } } }
I20260621 05:14:40.869249 24990 sys_catalog.cc:458] T 00000000000000000000000000000000 P c06fa43548cb4e63b2426a71b5b33741 [sys.catalog]: This master's current role is: LEADER
I20260621 05:14:40.869325 25001 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260621 05:14:40.869485 24988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:14:40.863461 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.16:59019 (local address 127.0.0.1:32903)
0621 05:14:40.863681 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0621 05:14:40.863691 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0621 05:14:40.863746 (+ 55us) server_negotiation.cc:407] Connection header received
0621 05:14:40.864269 (+ 523us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:14:40.864276 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:14:40.864327 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:14:40.864439 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:14:40.864699 (+ 260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:14:40.865677 (+ 978us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:14:40.866789 (+ 1112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:14:40.867332 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:14:40.867432 (+ 100us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:14:40.868283 (+ 851us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:14:40.868286 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:14:40.868287 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 05:14:40.868503 (+ 216us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:14:40.868547 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 05:14:40.868563 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:14:40.868650 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0621 05:14:40.868796 (+ 146us) server_negotiation.cc:299] Negotiation successful
0621 05:14:40.868824 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":164,"thread_start_us":90,"threads_started":1}
I20260621 05:14:40.870024 25001 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260621 05:14:40.870532 3598 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260621 05:14:40.871748 25001 catalog_manager.cc:1347] Generated new cluster ID: 9e0594db99894473a2c15c55edfb1aee
I20260621 05:14:40.871801 25001 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260621 05:14:40.875783 24988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:14:40.872118 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.16:53079 (local address 127.0.0.1:32903)
0621 05:14:40.872152 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0621 05:14:40.872154 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:14:40.872286 (+ 132us) server_negotiation.cc:407] Connection header received
0621 05:14:40.872348 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:14:40.872349 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:14:40.872359 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:14:40.872419 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:14:40.872555 (+ 136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:14:40.873272 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:14:40.874317 (+ 1045us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:14:40.874650 (+ 333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:14:40.874716 (+ 66us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:14:40.875531 (+ 815us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:14:40.875533 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:14:40.875533 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 05:14:40.875538 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:14:40.875554 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 05:14:40.875558 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:14:40.875610 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0621 05:14:40.875698 (+ 88us) server_negotiation.cc:299] Negotiation successful
0621 05:14:40.875707 (+ 9us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260621 05:14:40.887755 25001 catalog_manager.cc:1370] Generated new certificate authority record
I20260621 05:14:40.888309 25001 catalog_manager.cc:1504] Loading token signing keys...
I20260621 05:14:40.902074 25001 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c06fa43548cb4e63b2426a71b5b33741: Generated new TSK 0
I20260621 05:14:40.902333 25001 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260621 05:14:40.906386 3598 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWZOx2_/build/release/bin/kudu with pid 24916
W20260621 05:14:40.910871 24901 connection.cc:535] client connection to 127.0.0.1:32903 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:32903): Connection reset by peer (error 104)
2026-06-21T05:14:40Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1372 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1782018616648578-3598-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 21 05:14:40 dist-test-slave-vrq1 krb5kdc[25015](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 21 05:14:40 dist-test-slave-vrq1 krb5kdc[25015](info): set up 2 sockets
Jun 21 05:14:40 dist-test-slave-vrq1 krb5kdc[25015](info): commencing operation
krb5kdc: starting...
W20260621 05:14:41.952479 3598 mini_kdc.cc:121] Time spent starting KDC: real 1.027s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1782018616648578-3598-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1782018616648578-3598-0/krb5kdc/test-user.keytab.
Jun 21 05:14:42 dist-test-slave-vrq1 krb5kdc[25015](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782018882, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-21T05:14:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-21T05:14:42Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1782018616648578-3598-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1782018616648578-3598-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260621 05:14:42.082967 3598 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWZOx2_/build/release/bin/kudu
/tmp/dist-test-taskWZOx2_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1782018616648578-3598-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1782018616648578-3598-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1782018616648578-3598-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1782018616648578-3598-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38235
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1782018616648578-3598-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:43989
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:38235
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1782018616648578-3598-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1782018616648578-3598-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1782018616648578-3598-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 05:14:42.166199 25028 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 05:14:42.166486 25028 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 05:14:42.166529 25028 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 05:14:42.167806 25028 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 05:14:42.167882 25028 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 05:14:42.167907 25028 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 05:14:42.167936 25028 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 05:14:42.167966 25028 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 05:14:42.169553 25028 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:43989
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1782018616648578-3598-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1782018616648578-3598-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:38235
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1782018616648578-3598-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38235
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1782018616648578-3598-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25028
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1782018616648578-3598-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 21 Jun 2026 04:16:16 UTC on ccycloud.quasar-xycaha.root.comops.site
build id 4457881
I20260621 05:14:42.169845 25028 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 05:14:42.170085 25028 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 05:14:42.175243 25036 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:14:42.175356 25028 server_base.cc:1034] running on GCE node
W20260621 05:14:42.175287 25033 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:14:42.175225 25034 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:14:42.175736 25028 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 05:14:42.176050 25028 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 05:14:42.177253 25028 hybrid_clock.cc:648] HybridClock initialized: now 1782018882177221 us; error 54 us; skew 500 ppm
Jun 21 05:14:42 dist-test-slave-vrq1 krb5kdc[25015](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782018882, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 05:14:42.180056 25028 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260621 05:14:42.180581 25028 webserver.cc:460] Webserver started at http://127.0.0.1:33239/ using document root <none> and password file <none>
I20260621 05:14:42.180799 25028 fs_manager.cc:359] Metadata directory not provided
I20260621 05:14:42.180855 25028 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 05:14:42.180949 25028 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 05:14:42.182029 25028 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1782018616648578-3598-0/minicluster-data/master-0/data/instance:
uuid: "183647de1de84d19ac9762f2d2a00c02"
format_stamp: "Formatted at 2026-06-21 05:14:42 on dist-test-slave-vrq1"
server_key: "3f7462535698ae6996c7e6f7fae35c30"
server_key_iv: "d056aae916813a5645dbd66a6c6d6e17"
server_key_version: "encryptionkey@0"
I20260621 05:14:42.182338 25028 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1782018616648578-3598-0/minicluster-data/master-0/wal/instance:
uuid: "183647de1de84d19ac9762f2d2a00c02"
format_stamp: "Formatted at 2026-06-21 05:14:42 on dist-test-slave-vrq1"
server_key: "3f7462535698ae6996c7e6f7fae35c30"
server_key_iv: "d056aae916813a5645dbd66a6c6d6e17"
server_key_version: "encryptionkey@0"
I20260621 05:14:42.183833 25028 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260621 05:14:42.184746 25043 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 05:14:42.185057 25028 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260621 05:14:42.185191 25028 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1782018616648578-3598-0/minicluster-data/master-0/data,/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1782018616648578-3598-0/minicluster-data/master-0/wal
uuid: "183647de1de84d19ac9762f2d2a00c02"
format_stamp: "Formatted at 2026-06-21 05:14:42 on dist-test-slave-vrq1"
server_key: "3f7462535698ae6996c7e6f7fae35c30"
server_key_iv: "d056aae916813a5645dbd66a6c6d6e17"
server_key_version: "encryptionkey@0"
I20260621 05:14:42.185289 25028 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1782018616648578-3598-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1782018616648578-3598-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1782018616648578-3598-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 05:14:42.189071 25028 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 05:14:42.191431 25028 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 05:14:42.191601 25028 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 05:14:42.196211 25028 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:38235
I20260621 05:14:42.196285 25095 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:38235 every 8 connection(s)
I20260621 05:14:42.196671 25028 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1782018616648578-3598-0/minicluster-data/master-0/data/info.pb
I20260621 05:14:42.197726 25096 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 05:14:42.198318 3598 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWZOx2_/build/release/bin/kudu as pid 25028
I20260621 05:14:42.198441 3598 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1782018616648578-3598-0/minicluster-data/master-0/wal/instance
I20260621 05:14:42.198619 3598 external_mini_cluster.cc:1428] Setting key 155e48797cb28443bcedccddd0c9761a
I20260621 05:14:42.201259 25096 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 183647de1de84d19ac9762f2d2a00c02: Bootstrap starting.
I20260621 05:14:42.201707 25099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:14:42.199255 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.16:59495 (local address 127.0.0.1:38235)
0621 05:14:42.199489 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0621 05:14:42.199499 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0621 05:14:42.199536 (+ 37us) server_negotiation.cc:407] Connection header received
0621 05:14:42.200110 (+ 574us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:14:42.200119 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:14:42.200155 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:14:42.200326 (+ 171us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:14:42.201126 (+ 800us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:14:42.201130 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:14:42.201131 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 05:14:42.201135 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:14:42.201181 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 05:14:42.201195 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:14:42.201274 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0621 05:14:42.201436 (+ 162us) server_negotiation.cc:299] Negotiation successful
0621 05:14:42.201492 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":161,"thread_start_us":91,"threads_started":1}
I20260621 05:14:42.202112 25096 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 183647de1de84d19ac9762f2d2a00c02: Neither blocks nor log segments found. Creating new log.
I20260621 05:14:42.202462 25096 log.cc:824] T 00000000000000000000000000000000 P 183647de1de84d19ac9762f2d2a00c02: Log is configured to *not* fsync() on all Append() calls
I20260621 05:14:42.203130 3598 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260621 05:14:42.203474 25096 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 183647de1de84d19ac9762f2d2a00c02: No bootstrap required, opened a new log
I20260621 05:14:42.206151 25099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:14:42.204882 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.16:50737 (local address 127.0.0.1:38235)
0621 05:14:42.204929 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0621 05:14:42.204931 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:14:42.205008 (+ 77us) server_negotiation.cc:407] Connection header received
0621 05:14:42.205101 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:14:42.205102 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:14:42.205108 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:14:42.205164 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:14:42.205912 (+ 748us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:14:42.205914 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:14:42.205915 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 05:14:42.205919 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:14:42.205938 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 05:14:42.205944 (+ 6us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:14:42.206017 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0621 05:14:42.206066 (+ 49us) server_negotiation.cc:299] Negotiation successful
0621 05:14:42.206082 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260621 05:14:42.206424 25096 raft_consensus.cc:348] T 00000000000000000000000000000000 P 183647de1de84d19ac9762f2d2a00c02 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "183647de1de84d19ac9762f2d2a00c02" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38235 } }
I20260621 05:14:42.206641 25096 raft_consensus.cc:374] T 00000000000000000000000000000000 P 183647de1de84d19ac9762f2d2a00c02 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 05:14:42.206722 25096 raft_consensus.cc:729] T 00000000000000000000000000000000 P 183647de1de84d19ac9762f2d2a00c02 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 183647de1de84d19ac9762f2d2a00c02, State: Initialized, Role: FOLLOWER
I20260621 05:14:42.206859 25096 consensus_queue.cc:260] T 00000000000000000000000000000000 P 183647de1de84d19ac9762f2d2a00c02 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "183647de1de84d19ac9762f2d2a00c02" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38235 } }
I20260621 05:14:42.206974 25096 raft_consensus.cc:388] T 00000000000000000000000000000000 P 183647de1de84d19ac9762f2d2a00c02 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260621 05:14:42.207022 25096 raft_consensus.cc:482] T 00000000000000000000000000000000 P 183647de1de84d19ac9762f2d2a00c02 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260621 05:14:42.207072 25096 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 183647de1de84d19ac9762f2d2a00c02 [term 0 FOLLOWER]: Advancing to term 1
I20260621 05:14:42.207760 25096 raft_consensus.cc:504] T 00000000000000000000000000000000 P 183647de1de84d19ac9762f2d2a00c02 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "183647de1de84d19ac9762f2d2a00c02" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38235 } }
I20260621 05:14:42.207890 25096 leader_election.cc:302] T 00000000000000000000000000000000 P 183647de1de84d19ac9762f2d2a00c02 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 183647de1de84d19ac9762f2d2a00c02; no voters:
I20260621 05:14:42.208072 25096 leader_election.cc:288] T 00000000000000000000000000000000 P 183647de1de84d19ac9762f2d2a00c02 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260621 05:14:42.208119 25106 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 183647de1de84d19ac9762f2d2a00c02 [term 1 FOLLOWER]: Leader election won for term 1
I20260621 05:14:42.208318 25106 raft_consensus.cc:686] T 00000000000000000000000000000000 P 183647de1de84d19ac9762f2d2a00c02 [term 1 LEADER]: Becoming Leader. State: Replica: 183647de1de84d19ac9762f2d2a00c02, State: Running, Role: LEADER
I20260621 05:14:42.208352 25096 sys_catalog.cc:564] T 00000000000000000000000000000000 P 183647de1de84d19ac9762f2d2a00c02 [sys.catalog]: configured and running, proceeding with master startup.
I20260621 05:14:42.208467 25106 consensus_queue.cc:237] T 00000000000000000000000000000000 P 183647de1de84d19ac9762f2d2a00c02 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "183647de1de84d19ac9762f2d2a00c02" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38235 } }
I20260621 05:14:42.208894 25108 sys_catalog.cc:455] T 00000000000000000000000000000000 P 183647de1de84d19ac9762f2d2a00c02 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 183647de1de84d19ac9762f2d2a00c02. Latest consensus state: current_term: 1 leader_uuid: "183647de1de84d19ac9762f2d2a00c02" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "183647de1de84d19ac9762f2d2a00c02" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38235 } } }
I20260621 05:14:42.208884 25107 sys_catalog.cc:455] T 00000000000000000000000000000000 P 183647de1de84d19ac9762f2d2a00c02 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "183647de1de84d19ac9762f2d2a00c02" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "183647de1de84d19ac9762f2d2a00c02" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38235 } } }
I20260621 05:14:42.209015 25108 sys_catalog.cc:458] T 00000000000000000000000000000000 P 183647de1de84d19ac9762f2d2a00c02 [sys.catalog]: This master's current role is: LEADER
I20260621 05:14:42.209025 25107 sys_catalog.cc:458] T 00000000000000000000000000000000 P 183647de1de84d19ac9762f2d2a00c02 [sys.catalog]: This master's current role is: LEADER
I20260621 05:14:42.209244 25115 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260621 05:14:42.209784 25115 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260621 05:14:42.211504 25115 catalog_manager.cc:1347] Generated new cluster ID: f7511a45b51e447893bf6a93375073e7
I20260621 05:14:42.211580 25115 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260621 05:14:42.215890 25115 catalog_manager.cc:1370] Generated new certificate authority record
I20260621 05:14:42.216073 25115 catalog_manager.cc:1504] Loading token signing keys...
I20260621 05:14:42.223055 25115 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 183647de1de84d19ac9762f2d2a00c02: Generated new TSK 0
I20260621 05:14:42.223295 25115 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260621 05:14:42.227294 3598 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWZOx2_/build/release/bin/kudu with pid 25028
2026-06-21T05:14:42Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1320 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1782018616648578-3598-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 21 05:14:42 dist-test-slave-vrq1 krb5kdc[25127](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 21 05:14:42 dist-test-slave-vrq1 krb5kdc[25127](info): set up 2 sockets
Jun 21 05:14:42 dist-test-slave-vrq1 krb5kdc[25127](info): commencing operation
krb5kdc: starting...
W20260621 05:14:43.287909 3598 mini_kdc.cc:121] Time spent starting KDC: real 1.042s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1782018616648578-3598-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1782018616648578-3598-0/krb5kdc/test-user.keytab.
Jun 21 05:14:43 dist-test-slave-vrq1 krb5kdc[25127](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782018883, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-21T05:14:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-21T05:14:43Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.131.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.131.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.131.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1782018616648578-3598-0/krb5kdc/kudu_127.3.131.190.keytab.
Entry for principal kudu/127.3.131.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1782018616648578-3598-0/krb5kdc/kudu_127.3.131.190.keytab.
I20260621 05:14:43.427052 3598 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWZOx2_/build/release/bin/kudu
/tmp/dist-test-taskWZOx2_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1782018616648578-3598-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1782018616648578-3598-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1782018616648578-3598-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1782018616648578-3598-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.131.190:36873
--webserver_interface=127.3.131.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1782018616648578-3598-0/krb5kdc/kudu_127.3.131.190.keytab
--principal=kudu/127.3.131.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.131.148:34957
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.131.190:36873
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1782018616648578-3598-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1782018616648578-3598-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1782018616648578-3598-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 05:14:43.518020 25140 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 05:14:43.518241 25140 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 05:14:43.518267 25140 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 05:14:43.519574 25140 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 05:14:43.519627 25140 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 05:14:43.519641 25140 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 05:14:43.519654 25140 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 05:14:43.519667 25140 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 05:14:43.521853 25140 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.131.148:34957
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1782018616648578-3598-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1782018616648578-3598-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.131.190:36873
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1782018616648578-3598-0/krb5kdc/kudu_127.3.131.190.keytab
--principal=kudu/127.3.131.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.131.190:36873
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1782018616648578-3598-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.131.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25140
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1782018616648578-3598-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 21 Jun 2026 04:16:16 UTC on ccycloud.quasar-xycaha.root.comops.site
build id 4457881
I20260621 05:14:43.522292 25140 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 05:14:43.522600 25140 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 05:14:43.529618 25146 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:14:43.529789 25148 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:14:43.529647 25145 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:14:43.529844 25140 server_base.cc:1034] running on GCE node
I20260621 05:14:43.530206 25140 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 05:14:43.530651 25140 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 05:14:43.531838 25140 hybrid_clock.cc:648] HybridClock initialized: now 1782018883531813 us; error 36 us; skew 500 ppm
Jun 21 05:14:43 dist-test-slave-vrq1 krb5kdc[25127](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782018883, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.131.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 05:14:43.540536 25140 init.cc:377] Logged in from keytab as kudu/127.3.131.190@KRBTEST.COM (short username kudu)
I20260621 05:14:43.541067 25140 webserver.cc:460] Webserver started at http://127.3.131.190:40139/ using document root <none> and password file <none>
I20260621 05:14:43.541329 25140 fs_manager.cc:359] Metadata directory not provided
I20260621 05:14:43.541378 25140 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 05:14:43.541550 25140 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 05:14:43.542722 25140 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1782018616648578-3598-0/minicluster-data/master-0/data/instance:
uuid: "9e40d03177a048b58fe778351679f251"
format_stamp: "Formatted at 2026-06-21 05:14:43 on dist-test-slave-vrq1"
server_key: "ea460ba542cdea7792bd6b7ad7342c30"
server_key_iv: "63ce87d61d1cf19e5d47436d53ac3d15"
server_key_version: "encryptionkey@0"
I20260621 05:14:43.543107 25140 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1782018616648578-3598-0/minicluster-data/master-0/wal/instance:
uuid: "9e40d03177a048b58fe778351679f251"
format_stamp: "Formatted at 2026-06-21 05:14:43 on dist-test-slave-vrq1"
server_key: "ea460ba542cdea7792bd6b7ad7342c30"
server_key_iv: "63ce87d61d1cf19e5d47436d53ac3d15"
server_key_version: "encryptionkey@0"
I20260621 05:14:43.544667 25140 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260621 05:14:43.545595 25155 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 05:14:43.545864 25140 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260621 05:14:43.545971 25140 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1782018616648578-3598-0/minicluster-data/master-0/data,/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1782018616648578-3598-0/minicluster-data/master-0/wal
uuid: "9e40d03177a048b58fe778351679f251"
format_stamp: "Formatted at 2026-06-21 05:14:43 on dist-test-slave-vrq1"
server_key: "ea460ba542cdea7792bd6b7ad7342c30"
server_key_iv: "63ce87d61d1cf19e5d47436d53ac3d15"
server_key_version: "encryptionkey@0"
I20260621 05:14:43.546087 25140 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1782018616648578-3598-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1782018616648578-3598-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1782018616648578-3598-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 05:14:43.558418 25140 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 05:14:43.561254 25140 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 05:14:43.561437 25140 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 05:14:43.566046 25140 rpc_server.cc:307] RPC server started. Bound to: 127.3.131.190:36873
I20260621 05:14:43.566146 25207 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.131.190:36873 every 8 connection(s)
I20260621 05:14:43.566509 25140 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1782018616648578-3598-0/minicluster-data/master-0/data/info.pb
I20260621 05:14:43.567575 25208 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 05:14:43.570458 25208 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9e40d03177a048b58fe778351679f251: Bootstrap starting.
I20260621 05:14:43.571163 25208 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9e40d03177a048b58fe778351679f251: Neither blocks nor log segments found. Creating new log.
I20260621 05:14:43.571527 25208 log.cc:824] T 00000000000000000000000000000000 P 9e40d03177a048b58fe778351679f251: Log is configured to *not* fsync() on all Append() calls
I20260621 05:14:43.572324 25208 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9e40d03177a048b58fe778351679f251: No bootstrap required, opened a new log
I20260621 05:14:43.572892 3598 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWZOx2_/build/release/bin/kudu as pid 25140
I20260621 05:14:43.573027 3598 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1782018616648578-3598-0/minicluster-data/master-0/wal/instance
I20260621 05:14:43.573212 3598 external_mini_cluster.cc:1428] Setting key c06c218f68e7c05db8974150fd1e061a
I20260621 05:14:43.574568 25208 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9e40d03177a048b58fe778351679f251 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e40d03177a048b58fe778351679f251" member_type: VOTER last_known_addr { host: "127.3.131.190" port: 36873 } }
I20260621 05:14:43.574714 25208 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9e40d03177a048b58fe778351679f251 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 05:14:43.574736 25208 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9e40d03177a048b58fe778351679f251 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9e40d03177a048b58fe778351679f251, State: Initialized, Role: FOLLOWER
I20260621 05:14:43.574873 25208 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9e40d03177a048b58fe778351679f251 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e40d03177a048b58fe778351679f251" member_type: VOTER last_known_addr { host: "127.3.131.190" port: 36873 } }
I20260621 05:14:43.574965 25208 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9e40d03177a048b58fe778351679f251 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260621 05:14:43.574999 25208 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9e40d03177a048b58fe778351679f251 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260621 05:14:43.575044 25208 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9e40d03177a048b58fe778351679f251 [term 0 FOLLOWER]: Advancing to term 1
I20260621 05:14:43.576263 25208 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9e40d03177a048b58fe778351679f251 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e40d03177a048b58fe778351679f251" member_type: VOTER last_known_addr { host: "127.3.131.190" port: 36873 } }
I20260621 05:14:43.576365 25208 leader_election.cc:302] T 00000000000000000000000000000000 P 9e40d03177a048b58fe778351679f251 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9e40d03177a048b58fe778351679f251; no voters:
I20260621 05:14:43.576568 25208 leader_election.cc:288] T 00000000000000000000000000000000 P 9e40d03177a048b58fe778351679f251 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260621 05:14:43.576623 25213 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9e40d03177a048b58fe778351679f251 [term 1 FOLLOWER]: Leader election won for term 1
I20260621 05:14:43.576781 25213 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9e40d03177a048b58fe778351679f251 [term 1 LEADER]: Becoming Leader. State: Replica: 9e40d03177a048b58fe778351679f251, State: Running, Role: LEADER
I20260621 05:14:43.576870 25208 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9e40d03177a048b58fe778351679f251 [sys.catalog]: configured and running, proceeding with master startup.
I20260621 05:14:43.576923 25213 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9e40d03177a048b58fe778351679f251 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e40d03177a048b58fe778351679f251" member_type: VOTER last_known_addr { host: "127.3.131.190" port: 36873 } }
I20260621 05:14:43.577363 25213 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9e40d03177a048b58fe778351679f251 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9e40d03177a048b58fe778351679f251. Latest consensus state: current_term: 1 leader_uuid: "9e40d03177a048b58fe778351679f251" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e40d03177a048b58fe778351679f251" member_type: VOTER last_known_addr { host: "127.3.131.190" port: 36873 } } }
I20260621 05:14:43.577369 25215 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9e40d03177a048b58fe778351679f251 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9e40d03177a048b58fe778351679f251" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e40d03177a048b58fe778351679f251" member_type: VOTER last_known_addr { host: "127.3.131.190" port: 36873 } } }
I20260621 05:14:43.577517 25215 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9e40d03177a048b58fe778351679f251 [sys.catalog]: This master's current role is: LEADER
I20260621 05:14:43.577752 25213 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9e40d03177a048b58fe778351679f251 [sys.catalog]: This master's current role is: LEADER
I20260621 05:14:43.577934 25222 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260621 05:14:43.578481 25222 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260621 05:14:43.580248 25222 catalog_manager.cc:1347] Generated new cluster ID: 9c4784aecf40402bb3f8099da7d5ac03
I20260621 05:14:43.580303 25222 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Jun 21 05:14:43 dist-test-slave-vrq1 krb5kdc[25127](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782018883, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.131.190@KRBTEST.COM
I20260621 05:14:43.586489 25212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:14:43.573797 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55687 (local address 127.3.131.190:36873)
0621 05:14:43.574013 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0621 05:14:43.574023 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0621 05:14:43.574057 (+ 34us) server_negotiation.cc:407] Connection header received
0621 05:14:43.574603 (+ 546us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:14:43.574611 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:14:43.574669 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:14:43.574885 (+ 216us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:14:43.575853 (+ 968us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:14:43.577127 (+ 1274us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:14:43.578338 (+ 1211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:14:43.579143 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:14:43.579310 (+ 167us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:14:43.582288 (+ 2978us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:14:43.582292 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:14:43.582293 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:14:43.582307 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:14:43.584447 (+ 2140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:14:43.584956 (+ 509us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:14:43.584959 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:14:43.584960 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:14:43.585122 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:14:43.585370 (+ 248us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:14:43.585371 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:14:43.585371 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:14:43.585734 (+ 363us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:14:43.585921 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0621 05:14:43.586053 (+ 132us) server_negotiation.cc:299] Negotiation successful
0621 05:14:43.586191 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":149,"thread_start_us":86,"threads_started":1}
I20260621 05:14:43.587973 3598 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260621 05:14:43.600433 25212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:14:43.589518 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46719 (local address 127.3.131.190:36873)
0621 05:14:43.589557 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0621 05:14:43.589559 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:14:43.591406 (+ 1847us) server_negotiation.cc:407] Connection header received
0621 05:14:43.591491 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:14:43.591492 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:14:43.591505 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:14:43.591575 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:14:43.592660 (+ 1085us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:14:43.593478 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:14:43.594592 (+ 1114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:14:43.594923 (+ 331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:14:43.594990 (+ 67us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:14:43.596680 (+ 1690us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:14:43.596682 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:14:43.596682 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:14:43.596691 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:14:43.598452 (+ 1761us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:14:43.599214 (+ 762us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:14:43.599215 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:14:43.599215 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:14:43.599282 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:14:43.599777 (+ 495us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:14:43.599779 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:14:43.599779 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:14:43.599936 (+ 157us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:14:43.600065 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0621 05:14:43.600254 (+ 189us) server_negotiation.cc:299] Negotiation successful
0621 05:14:43.600332 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260621 05:14:43.626020 25222 catalog_manager.cc:1370] Generated new certificate authority record
I20260621 05:14:43.626813 25222 catalog_manager.cc:1504] Loading token signing keys...
I20260621 05:14:43.634454 25222 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9e40d03177a048b58fe778351679f251: Generated new TSK 0
I20260621 05:14:43.635113 25222 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260621 05:14:43.642549 3598 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWZOx2_/build/release/bin/kudu with pid 25140
W20260621 05:14:43.649274 25125 connection.cc:535] client connection to 127.3.131.190:36873 recv error: Network error: failed to read from TLS socket (remote: 127.3.131.190:36873): Connection reset by peer (error 104)
2026-06-21T05:14:43Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (1417 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1782018616648578-3598-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 21 05:14:43 dist-test-slave-vrq1 krb5kdc[25239](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 21 05:14:43 dist-test-slave-vrq1 krb5kdc[25239](info): set up 2 sockets
Jun 21 05:14:43 dist-test-slave-vrq1 krb5kdc[25239](info): commencing operation
krb5kdc: starting...
W20260621 05:14:44.707774 3598 mini_kdc.cc:121] Time spent starting KDC: real 1.044s user 0.006s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1782018616648578-3598-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1782018616648578-3598-0/krb5kdc/test-user.keytab.
Jun 21 05:14:44 dist-test-slave-vrq1 krb5kdc[25239](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782018884, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-21T05:14:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-21T05:14:44Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.131.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.131.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.131.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1782018616648578-3598-0/krb5kdc/kudu_127.3.131.190.keytab.
Entry for principal kudu/127.3.131.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1782018616648578-3598-0/krb5kdc/kudu_127.3.131.190.keytab.
I20260621 05:14:44.842823 3598 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWZOx2_/build/release/bin/kudu
/tmp/dist-test-taskWZOx2_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1782018616648578-3598-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1782018616648578-3598-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1782018616648578-3598-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1782018616648578-3598-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.131.190:34447
--webserver_interface=127.3.131.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1782018616648578-3598-0/krb5kdc/kudu_127.3.131.190.keytab
--principal=kudu/127.3.131.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.131.148:33285
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.131.190:34447
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1782018616648578-3598-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1782018616648578-3598-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1782018616648578-3598-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 05:14:44.924826 25252 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 05:14:44.925114 25252 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 05:14:44.925154 25252 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 05:14:44.926712 25252 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 05:14:44.926831 25252 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 05:14:44.926860 25252 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 05:14:44.926882 25252 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 05:14:44.926914 25252 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 05:14:44.928833 25252 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.131.148:33285
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1782018616648578-3598-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1782018616648578-3598-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.131.190:34447
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1782018616648578-3598-0/krb5kdc/kudu_127.3.131.190.keytab
--principal=kudu/127.3.131.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.131.190:34447
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1782018616648578-3598-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.131.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25252
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1782018616648578-3598-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 21 Jun 2026 04:16:16 UTC on ccycloud.quasar-xycaha.root.comops.site
build id 4457881
I20260621 05:14:44.929177 25252 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 05:14:44.929442 25252 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260621 05:14:44.935137 25252 server_base.cc:1034] running on GCE node
W20260621 05:14:44.935077 25257 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:14:44.935078 25260 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:14:44.935101 25258 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:14:44.935704 25252 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 05:14:44.936038 25252 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 05:14:44.937356 25252 hybrid_clock.cc:648] HybridClock initialized: now 1782018884937335 us; error 32 us; skew 500 ppm
Jun 21 05:14:44 dist-test-slave-vrq1 krb5kdc[25239](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782018884, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.131.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 05:14:44.940666 25252 init.cc:377] Logged in from keytab as kudu/127.3.131.190@KRBTEST.COM (short username kudu)
I20260621 05:14:44.941193 25252 webserver.cc:460] Webserver started at http://127.3.131.190:43251/ using document root <none> and password file <none>
I20260621 05:14:44.941412 25252 fs_manager.cc:359] Metadata directory not provided
I20260621 05:14:44.941455 25252 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 05:14:44.941550 25252 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 05:14:44.942646 25252 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1782018616648578-3598-0/minicluster-data/master-0/data/instance:
uuid: "52dfc2abeb184562b7a88a79526201dc"
format_stamp: "Formatted at 2026-06-21 05:14:44 on dist-test-slave-vrq1"
server_key: "0c22fd531478520993159593036fcbaa"
server_key_iv: "6602b9a8df3736aa0e2238bb65dd112f"
server_key_version: "encryptionkey@0"
I20260621 05:14:44.942958 25252 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1782018616648578-3598-0/minicluster-data/master-0/wal/instance:
uuid: "52dfc2abeb184562b7a88a79526201dc"
format_stamp: "Formatted at 2026-06-21 05:14:44 on dist-test-slave-vrq1"
server_key: "0c22fd531478520993159593036fcbaa"
server_key_iv: "6602b9a8df3736aa0e2238bb65dd112f"
server_key_version: "encryptionkey@0"
I20260621 05:14:44.944391 25252 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260621 05:14:44.945117 25267 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 05:14:44.945363 25252 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260621 05:14:44.945467 25252 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1782018616648578-3598-0/minicluster-data/master-0/data,/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1782018616648578-3598-0/minicluster-data/master-0/wal
uuid: "52dfc2abeb184562b7a88a79526201dc"
format_stamp: "Formatted at 2026-06-21 05:14:44 on dist-test-slave-vrq1"
server_key: "0c22fd531478520993159593036fcbaa"
server_key_iv: "6602b9a8df3736aa0e2238bb65dd112f"
server_key_version: "encryptionkey@0"
I20260621 05:14:44.945552 25252 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1782018616648578-3598-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1782018616648578-3598-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1782018616648578-3598-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 05:14:44.963603 25252 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 05:14:44.966147 25252 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 05:14:44.966327 25252 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 05:14:44.970546 25252 rpc_server.cc:307] RPC server started. Bound to: 127.3.131.190:34447
I20260621 05:14:44.970587 25319 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.131.190:34447 every 8 connection(s)
I20260621 05:14:44.970994 25252 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1782018616648578-3598-0/minicluster-data/master-0/data/info.pb
I20260621 05:14:44.971951 25320 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 05:14:44.974403 25320 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 52dfc2abeb184562b7a88a79526201dc: Bootstrap starting.
I20260621 05:14:44.975088 25320 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 52dfc2abeb184562b7a88a79526201dc: Neither blocks nor log segments found. Creating new log.
I20260621 05:14:44.975447 25320 log.cc:824] T 00000000000000000000000000000000 P 52dfc2abeb184562b7a88a79526201dc: Log is configured to *not* fsync() on all Append() calls
I20260621 05:14:44.976218 25320 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 52dfc2abeb184562b7a88a79526201dc: No bootstrap required, opened a new log
I20260621 05:14:44.978047 25320 raft_consensus.cc:348] T 00000000000000000000000000000000 P 52dfc2abeb184562b7a88a79526201dc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "52dfc2abeb184562b7a88a79526201dc" member_type: VOTER last_known_addr { host: "127.3.131.190" port: 34447 } }
I20260621 05:14:44.978197 25320 raft_consensus.cc:374] T 00000000000000000000000000000000 P 52dfc2abeb184562b7a88a79526201dc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 05:14:44.978220 25320 raft_consensus.cc:729] T 00000000000000000000000000000000 P 52dfc2abeb184562b7a88a79526201dc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 52dfc2abeb184562b7a88a79526201dc, State: Initialized, Role: FOLLOWER
I20260621 05:14:44.978386 25320 consensus_queue.cc:260] T 00000000000000000000000000000000 P 52dfc2abeb184562b7a88a79526201dc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "52dfc2abeb184562b7a88a79526201dc" member_type: VOTER last_known_addr { host: "127.3.131.190" port: 34447 } }
I20260621 05:14:44.978472 25320 raft_consensus.cc:388] T 00000000000000000000000000000000 P 52dfc2abeb184562b7a88a79526201dc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260621 05:14:44.978511 25320 raft_consensus.cc:482] T 00000000000000000000000000000000 P 52dfc2abeb184562b7a88a79526201dc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260621 05:14:44.978591 25320 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 52dfc2abeb184562b7a88a79526201dc [term 0 FOLLOWER]: Advancing to term 1
I20260621 05:14:44.978585 3598 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWZOx2_/build/release/bin/kudu as pid 25252
I20260621 05:14:44.978734 3598 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1782018616648578-3598-0/minicluster-data/master-0/wal/instance
I20260621 05:14:44.978925 3598 external_mini_cluster.cc:1428] Setting key 2608d7793e527823b93fbfb92945e180
I20260621 05:14:44.979790 25320 raft_consensus.cc:504] T 00000000000000000000000000000000 P 52dfc2abeb184562b7a88a79526201dc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "52dfc2abeb184562b7a88a79526201dc" member_type: VOTER last_known_addr { host: "127.3.131.190" port: 34447 } }
I20260621 05:14:44.979919 25320 leader_election.cc:302] T 00000000000000000000000000000000 P 52dfc2abeb184562b7a88a79526201dc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 52dfc2abeb184562b7a88a79526201dc; no voters:
I20260621 05:14:44.980088 25320 leader_election.cc:288] T 00000000000000000000000000000000 P 52dfc2abeb184562b7a88a79526201dc [CANDIDATE]: Term 1 election: Requested vote from peers
I20260621 05:14:44.980118 25325 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 52dfc2abeb184562b7a88a79526201dc [term 1 FOLLOWER]: Leader election won for term 1
I20260621 05:14:44.980276 25325 raft_consensus.cc:686] T 00000000000000000000000000000000 P 52dfc2abeb184562b7a88a79526201dc [term 1 LEADER]: Becoming Leader. State: Replica: 52dfc2abeb184562b7a88a79526201dc, State: Running, Role: LEADER
I20260621 05:14:44.980413 25320 sys_catalog.cc:564] T 00000000000000000000000000000000 P 52dfc2abeb184562b7a88a79526201dc [sys.catalog]: configured and running, proceeding with master startup.
I20260621 05:14:44.980406 25325 consensus_queue.cc:237] T 00000000000000000000000000000000 P 52dfc2abeb184562b7a88a79526201dc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "52dfc2abeb184562b7a88a79526201dc" member_type: VOTER last_known_addr { host: "127.3.131.190" port: 34447 } }
I20260621 05:14:44.980891 25327 sys_catalog.cc:455] T 00000000000000000000000000000000 P 52dfc2abeb184562b7a88a79526201dc [sys.catalog]: SysCatalogTable state changed. Reason: New leader 52dfc2abeb184562b7a88a79526201dc. Latest consensus state: current_term: 1 leader_uuid: "52dfc2abeb184562b7a88a79526201dc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "52dfc2abeb184562b7a88a79526201dc" member_type: VOTER last_known_addr { host: "127.3.131.190" port: 34447 } } }
I20260621 05:14:44.980989 25327 sys_catalog.cc:458] T 00000000000000000000000000000000 P 52dfc2abeb184562b7a88a79526201dc [sys.catalog]: This master's current role is: LEADER
I20260621 05:14:44.981225 25326 sys_catalog.cc:455] T 00000000000000000000000000000000 P 52dfc2abeb184562b7a88a79526201dc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "52dfc2abeb184562b7a88a79526201dc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "52dfc2abeb184562b7a88a79526201dc" member_type: VOTER last_known_addr { host: "127.3.131.190" port: 34447 } } }
I20260621 05:14:44.981545 25326 sys_catalog.cc:458] T 00000000000000000000000000000000 P 52dfc2abeb184562b7a88a79526201dc [sys.catalog]: This master's current role is: LEADER
I20260621 05:14:44.981688 25333 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260621 05:14:44.982236 25333 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260621 05:14:44.983904 25333 catalog_manager.cc:1347] Generated new cluster ID: 98e7ac6370f3497886aaa5b3a994d47a
I20260621 05:14:44.983966 25333 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260621 05:14:44.985008 25324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:14:44.979477 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48257 (local address 127.3.131.190:34447)
0621 05:14:44.979693 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0621 05:14:44.979705 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0621 05:14:44.979741 (+ 36us) server_negotiation.cc:407] Connection header received
0621 05:14:44.979851 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:14:44.979857 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:14:44.979917 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:14:44.980052 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:14:44.980289 (+ 237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:14:44.981684 (+ 1395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:14:44.982675 (+ 991us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:14:44.983274 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:14:44.983378 (+ 104us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:14:44.984332 (+ 954us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:14:44.984336 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:14:44.984336 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 05:14:44.984508 (+ 172us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:14:44.984538 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 05:14:44.984548 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:14:44.984640 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0621 05:14:44.984728 (+ 88us) server_negotiation.cc:299] Negotiation successful
0621 05:14:44.984845 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":154,"thread_start_us":102,"threads_started":1}
I20260621 05:14:44.985988 3598 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260621 05:14:44.990487 25324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:14:44.987334 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42099 (local address 127.3.131.190:34447)
0621 05:14:44.987378 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0621 05:14:44.987380 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:14:44.987543 (+ 163us) server_negotiation.cc:407] Connection header received
0621 05:14:44.987598 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:14:44.987599 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:14:44.987610 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:14:44.987664 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:14:44.987806 (+ 142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:14:44.988597 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:14:44.989299 (+ 702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:14:44.989529 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:14:44.989602 (+ 73us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:14:44.990259 (+ 657us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:14:44.990261 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:14:44.990261 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 05:14:44.990265 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:14:44.990279 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 05:14:44.990283 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:14:44.990335 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0621 05:14:44.990400 (+ 65us) server_negotiation.cc:299] Negotiation successful
0621 05:14:44.990412 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260621 05:14:44.999547 25333 catalog_manager.cc:1370] Generated new certificate authority record
I20260621 05:14:45.000350 25333 catalog_manager.cc:1504] Loading token signing keys...
I20260621 05:14:45.013427 25333 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 52dfc2abeb184562b7a88a79526201dc: Generated new TSK 0
I20260621 05:14:45.013671 25333 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260621 05:14:45.022033 3598 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWZOx2_/build/release/bin/kudu with pid 25252
W20260621 05:14:45.028401 25237 connection.cc:535] client connection to 127.3.131.190:34447 recv error: Network error: failed to read from TLS socket (remote: 127.3.131.190:34447): Connection reset by peer (error 104)
2026-06-21T05:14:45Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1378 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1782018616648578-3598-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 21 05:14:45 dist-test-slave-vrq1 krb5kdc[25351](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 21 05:14:45 dist-test-slave-vrq1 krb5kdc[25351](info): set up 2 sockets
Jun 21 05:14:45 dist-test-slave-vrq1 krb5kdc[25351](info): commencing operation
krb5kdc: starting...
W20260621 05:14:46.119666 3598 mini_kdc.cc:121] Time spent starting KDC: real 1.078s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1782018616648578-3598-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1782018616648578-3598-0/krb5kdc/test-user.keytab.
Jun 21 05:14:46 dist-test-slave-vrq1 krb5kdc[25351](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782018886, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-21T05:14:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-21T05:14:46Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.131.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.131.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.131.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1782018616648578-3598-0/krb5kdc/kudu_127.3.131.190.keytab.
Entry for principal kudu/127.3.131.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1782018616648578-3598-0/krb5kdc/kudu_127.3.131.190.keytab.
I20260621 05:14:46.250819 3598 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWZOx2_/build/release/bin/kudu
/tmp/dist-test-taskWZOx2_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1782018616648578-3598-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1782018616648578-3598-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1782018616648578-3598-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1782018616648578-3598-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.131.190:43253
--webserver_interface=127.3.131.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1782018616648578-3598-0/krb5kdc/kudu_127.3.131.190.keytab
--principal=kudu/127.3.131.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.131.148:39149
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.131.190:43253
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1782018616648578-3598-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1782018616648578-3598-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1782018616648578-3598-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 05:14:46.341959 25364 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 05:14:46.342228 25364 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 05:14:46.342267 25364 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 05:14:46.343725 25364 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 05:14:46.343837 25364 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 05:14:46.343871 25364 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 05:14:46.343912 25364 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 05:14:46.343948 25364 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 05:14:46.346030 25364 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.131.148:39149
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1782018616648578-3598-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1782018616648578-3598-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.131.190:43253
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1782018616648578-3598-0/krb5kdc/kudu_127.3.131.190.keytab
--principal=kudu/127.3.131.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.131.190:43253
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1782018616648578-3598-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.131.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25364
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1782018616648578-3598-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 21 Jun 2026 04:16:16 UTC on ccycloud.quasar-xycaha.root.comops.site
build id 4457881
I20260621 05:14:46.346400 25364 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 05:14:46.346689 25364 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 05:14:46.352761 25369 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:14:46.352708 25370 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:14:46.352950 25364 server_base.cc:1034] running on GCE node
W20260621 05:14:46.352694 25372 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:14:46.353430 25364 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 05:14:46.353778 25364 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 05:14:46.354961 25364 hybrid_clock.cc:648] HybridClock initialized: now 1782018886354956 us; error 40 us; skew 500 ppm
Jun 21 05:14:46 dist-test-slave-vrq1 krb5kdc[25351](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782018886, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.131.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 05:14:46.357873 25364 init.cc:377] Logged in from keytab as kudu/127.3.131.190@KRBTEST.COM (short username kudu)
I20260621 05:14:46.358466 25364 webserver.cc:460] Webserver started at http://127.3.131.190:37031/ using document root <none> and password file <none>
I20260621 05:14:46.358712 25364 fs_manager.cc:359] Metadata directory not provided
I20260621 05:14:46.358783 25364 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 05:14:46.358950 25364 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 05:14:46.360132 25364 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1782018616648578-3598-0/minicluster-data/master-0/data/instance:
uuid: "a4ccdc9badb44ebb929949051b0dcf3f"
format_stamp: "Formatted at 2026-06-21 05:14:46 on dist-test-slave-vrq1"
server_key: "24bb205afffc5d4e59228e4586e861f3"
server_key_iv: "5da8357c0821ae1a201af26f827c8e8d"
server_key_version: "encryptionkey@0"
I20260621 05:14:46.360526 25364 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1782018616648578-3598-0/minicluster-data/master-0/wal/instance:
uuid: "a4ccdc9badb44ebb929949051b0dcf3f"
format_stamp: "Formatted at 2026-06-21 05:14:46 on dist-test-slave-vrq1"
server_key: "24bb205afffc5d4e59228e4586e861f3"
server_key_iv: "5da8357c0821ae1a201af26f827c8e8d"
server_key_version: "encryptionkey@0"
I20260621 05:14:46.361996 25364 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260621 05:14:46.362879 25379 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 05:14:46.363080 25364 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260621 05:14:46.363232 25364 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1782018616648578-3598-0/minicluster-data/master-0/data,/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1782018616648578-3598-0/minicluster-data/master-0/wal
uuid: "a4ccdc9badb44ebb929949051b0dcf3f"
format_stamp: "Formatted at 2026-06-21 05:14:46 on dist-test-slave-vrq1"
server_key: "24bb205afffc5d4e59228e4586e861f3"
server_key_iv: "5da8357c0821ae1a201af26f827c8e8d"
server_key_version: "encryptionkey@0"
I20260621 05:14:46.363318 25364 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1782018616648578-3598-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1782018616648578-3598-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1782018616648578-3598-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 05:14:46.366636 25364 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 05:14:46.369005 25364 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 05:14:46.369170 25364 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 05:14:46.373641 25364 rpc_server.cc:307] RPC server started. Bound to: 127.3.131.190:43253
I20260621 05:14:46.373714 25431 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.131.190:43253 every 8 connection(s)
I20260621 05:14:46.374080 25364 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1782018616648578-3598-0/minicluster-data/master-0/data/info.pb
I20260621 05:14:46.375349 25432 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 05:14:46.376475 3598 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWZOx2_/build/release/bin/kudu as pid 25364
I20260621 05:14:46.376592 3598 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1782018616648578-3598-0/minicluster-data/master-0/wal/instance
I20260621 05:14:46.376783 3598 external_mini_cluster.cc:1428] Setting key 0e910a70d5d677647308a46facc24bd9
I20260621 05:14:46.378014 25432 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a4ccdc9badb44ebb929949051b0dcf3f: Bootstrap starting.
I20260621 05:14:46.378782 25432 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a4ccdc9badb44ebb929949051b0dcf3f: Neither blocks nor log segments found. Creating new log.
I20260621 05:14:46.379285 25432 log.cc:824] T 00000000000000000000000000000000 P a4ccdc9badb44ebb929949051b0dcf3f: Log is configured to *not* fsync() on all Append() calls
I20260621 05:14:46.379557 25435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:14:46.377348 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55401 (local address 127.3.131.190:43253)
0621 05:14:46.377529 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0621 05:14:46.377541 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0621 05:14:46.377571 (+ 30us) server_negotiation.cc:407] Connection header received
0621 05:14:46.377971 (+ 400us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:14:46.377978 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:14:46.378003 (+ 25us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:14:46.378170 (+ 167us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:14:46.378911 (+ 741us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:14:46.378915 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:14:46.378916 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 05:14:46.378922 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:14:46.378966 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 05:14:46.378977 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:14:46.379049 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0621 05:14:46.379266 (+ 217us) server_negotiation.cc:299] Negotiation successful
0621 05:14:46.379319 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":131,"thread_start_us":91,"threads_started":1}
I20260621 05:14:46.380017 25432 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a4ccdc9badb44ebb929949051b0dcf3f: No bootstrap required, opened a new log
I20260621 05:14:46.380784 3598 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260621 05:14:46.382153 25432 raft_consensus.cc:348] T 00000000000000000000000000000000 P a4ccdc9badb44ebb929949051b0dcf3f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a4ccdc9badb44ebb929949051b0dcf3f" member_type: VOTER last_known_addr { host: "127.3.131.190" port: 43253 } }
I20260621 05:14:46.382349 25432 raft_consensus.cc:374] T 00000000000000000000000000000000 P a4ccdc9badb44ebb929949051b0dcf3f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 05:14:46.382401 25432 raft_consensus.cc:729] T 00000000000000000000000000000000 P a4ccdc9badb44ebb929949051b0dcf3f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a4ccdc9badb44ebb929949051b0dcf3f, State: Initialized, Role: FOLLOWER
I20260621 05:14:46.382520 25432 consensus_queue.cc:260] T 00000000000000000000000000000000 P a4ccdc9badb44ebb929949051b0dcf3f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a4ccdc9badb44ebb929949051b0dcf3f" member_type: VOTER last_known_addr { host: "127.3.131.190" port: 43253 } }
I20260621 05:14:46.382622 25432 raft_consensus.cc:388] T 00000000000000000000000000000000 P a4ccdc9badb44ebb929949051b0dcf3f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260621 05:14:46.382754 25432 raft_consensus.cc:482] T 00000000000000000000000000000000 P a4ccdc9badb44ebb929949051b0dcf3f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260621 05:14:46.382840 25432 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a4ccdc9badb44ebb929949051b0dcf3f [term 0 FOLLOWER]: Advancing to term 1
I20260621 05:14:46.383579 25432 raft_consensus.cc:504] T 00000000000000000000000000000000 P a4ccdc9badb44ebb929949051b0dcf3f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a4ccdc9badb44ebb929949051b0dcf3f" member_type: VOTER last_known_addr { host: "127.3.131.190" port: 43253 } }
I20260621 05:14:46.383694 25432 leader_election.cc:302] T 00000000000000000000000000000000 P a4ccdc9badb44ebb929949051b0dcf3f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a4ccdc9badb44ebb929949051b0dcf3f; no voters:
I20260621 05:14:46.383853 25432 leader_election.cc:288] T 00000000000000000000000000000000 P a4ccdc9badb44ebb929949051b0dcf3f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260621 05:14:46.383885 25442 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a4ccdc9badb44ebb929949051b0dcf3f [term 1 FOLLOWER]: Leader election won for term 1
I20260621 05:14:46.384095 25435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:14:46.382712 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32905 (local address 127.3.131.190:43253)
0621 05:14:46.382753 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0621 05:14:46.382755 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:14:46.382962 (+ 207us) server_negotiation.cc:407] Connection header received
0621 05:14:46.382984 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:14:46.382985 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:14:46.382989 (+ 4us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:14:46.383050 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:14:46.383730 (+ 680us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:14:46.383731 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:14:46.383732 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 05:14:46.383736 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:14:46.383751 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 05:14:46.383756 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:14:46.383814 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0621 05:14:46.383885 (+ 71us) server_negotiation.cc:299] Negotiation successful
0621 05:14:46.383900 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260621 05:14:46.384101 25442 raft_consensus.cc:686] T 00000000000000000000000000000000 P a4ccdc9badb44ebb929949051b0dcf3f [term 1 LEADER]: Becoming Leader. State: Replica: a4ccdc9badb44ebb929949051b0dcf3f, State: Running, Role: LEADER
I20260621 05:14:46.384152 25432 sys_catalog.cc:564] T 00000000000000000000000000000000 P a4ccdc9badb44ebb929949051b0dcf3f [sys.catalog]: configured and running, proceeding with master startup.
I20260621 05:14:46.384330 25442 consensus_queue.cc:237] T 00000000000000000000000000000000 P a4ccdc9badb44ebb929949051b0dcf3f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a4ccdc9badb44ebb929949051b0dcf3f" member_type: VOTER last_known_addr { host: "127.3.131.190" port: 43253 } }
I20260621 05:14:46.384758 25443 sys_catalog.cc:455] T 00000000000000000000000000000000 P a4ccdc9badb44ebb929949051b0dcf3f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a4ccdc9badb44ebb929949051b0dcf3f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a4ccdc9badb44ebb929949051b0dcf3f" member_type: VOTER last_known_addr { host: "127.3.131.190" port: 43253 } } }
I20260621 05:14:46.384896 25443 sys_catalog.cc:458] T 00000000000000000000000000000000 P a4ccdc9badb44ebb929949051b0dcf3f [sys.catalog]: This master's current role is: LEADER
I20260621 05:14:46.385169 25450 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260621 05:14:46.385706 25450 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260621 05:14:46.385911 25444 sys_catalog.cc:455] T 00000000000000000000000000000000 P a4ccdc9badb44ebb929949051b0dcf3f [sys.catalog]: SysCatalogTable state changed. Reason: New leader a4ccdc9badb44ebb929949051b0dcf3f. Latest consensus state: current_term: 1 leader_uuid: "a4ccdc9badb44ebb929949051b0dcf3f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a4ccdc9badb44ebb929949051b0dcf3f" member_type: VOTER last_known_addr { host: "127.3.131.190" port: 43253 } } }
I20260621 05:14:46.386101 25444 sys_catalog.cc:458] T 00000000000000000000000000000000 P a4ccdc9badb44ebb929949051b0dcf3f [sys.catalog]: This master's current role is: LEADER
I20260621 05:14:46.387629 25450 catalog_manager.cc:1347] Generated new cluster ID: 97c6ec512964464eb66736fa6d929ddb
I20260621 05:14:46.387702 25450 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260621 05:14:46.407765 25450 catalog_manager.cc:1370] Generated new certificate authority record
I20260621 05:14:46.407963 25450 catalog_manager.cc:1504] Loading token signing keys...
I20260621 05:14:46.411208 25450 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a4ccdc9badb44ebb929949051b0dcf3f: Generated new TSK 0
I20260621 05:14:46.411417 25450 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260621 05:14:46.416952 3598 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWZOx2_/build/release/bin/kudu with pid 25364
2026-06-21T05:14:46Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1393 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1782018616648578-3598-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 21 05:14:46 dist-test-slave-vrq1 krb5kdc[25463](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 21 05:14:46 dist-test-slave-vrq1 krb5kdc[25463](info): set up 2 sockets
Jun 21 05:14:46 dist-test-slave-vrq1 krb5kdc[25463](info): commencing operation
krb5kdc: starting...
W20260621 05:14:47.484447 3598 mini_kdc.cc:121] Time spent starting KDC: real 1.049s user 0.002s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1782018616648578-3598-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1782018616648578-3598-0/krb5kdc/test-user.keytab.
Jun 21 05:14:47 dist-test-slave-vrq1 krb5kdc[25463](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782018887, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-21T05:14:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-21T05:14:47Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.131.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.131.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.131.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1782018616648578-3598-0/krb5kdc/kudu_127.3.131.190.keytab.
Entry for principal kudu/127.3.131.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1782018616648578-3598-0/krb5kdc/kudu_127.3.131.190.keytab.
I20260621 05:14:47.623091 3598 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWZOx2_/build/release/bin/kudu
/tmp/dist-test-taskWZOx2_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1782018616648578-3598-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1782018616648578-3598-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1782018616648578-3598-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1782018616648578-3598-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.131.190:32973
--webserver_interface=127.3.131.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1782018616648578-3598-0/krb5kdc/kudu_127.3.131.190.keytab
--principal=kudu/127.3.131.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.131.148:41343
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.131.190:32973
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1782018616648578-3598-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1782018616648578-3598-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1782018616648578-3598-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 05:14:47.709872 25476 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 05:14:47.710148 25476 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 05:14:47.710177 25476 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 05:14:47.711529 25476 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 05:14:47.711580 25476 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 05:14:47.711596 25476 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 05:14:47.711609 25476 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 05:14:47.711621 25476 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 05:14:47.713306 25476 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.131.148:41343
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1782018616648578-3598-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1782018616648578-3598-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.131.190:32973
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1782018616648578-3598-0/krb5kdc/kudu_127.3.131.190.keytab
--principal=kudu/127.3.131.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.131.190:32973
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1782018616648578-3598-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.131.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25476
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1782018616648578-3598-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 21 Jun 2026 04:16:16 UTC on ccycloud.quasar-xycaha.root.comops.site
build id 4457881
I20260621 05:14:47.713558 25476 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 05:14:47.713757 25476 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 05:14:47.719727 25481 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:14:47.719727 25484 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:14:47.719758 25476 server_base.cc:1034] running on GCE node
W20260621 05:14:47.719738 25482 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:14:47.720207 25476 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 05:14:47.720544 25476 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 05:14:47.721772 25476 hybrid_clock.cc:648] HybridClock initialized: now 1782018887721758 us; error 55 us; skew 500 ppm
Jun 21 05:14:47 dist-test-slave-vrq1 krb5kdc[25463](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782018887, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.131.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 05:14:47.724691 25476 init.cc:377] Logged in from keytab as kudu/127.3.131.190@KRBTEST.COM (short username kudu)
I20260621 05:14:47.725266 25476 webserver.cc:460] Webserver started at http://127.3.131.190:37343/ using document root <none> and password file <none>
I20260621 05:14:47.725517 25476 fs_manager.cc:359] Metadata directory not provided
I20260621 05:14:47.725589 25476 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 05:14:47.725740 25476 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 05:14:47.726889 25476 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1782018616648578-3598-0/minicluster-data/master-0/data/instance:
uuid: "08a1e3b345fe4c12b2653730a6f2d4c3"
format_stamp: "Formatted at 2026-06-21 05:14:47 on dist-test-slave-vrq1"
server_key: "60fb2f17f5bb7f1293df95613b48ab7b"
server_key_iv: "815221c44d413f09dc1ac6b25379786c"
server_key_version: "encryptionkey@0"
I20260621 05:14:47.727344 25476 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1782018616648578-3598-0/minicluster-data/master-0/wal/instance:
uuid: "08a1e3b345fe4c12b2653730a6f2d4c3"
format_stamp: "Formatted at 2026-06-21 05:14:47 on dist-test-slave-vrq1"
server_key: "60fb2f17f5bb7f1293df95613b48ab7b"
server_key_iv: "815221c44d413f09dc1ac6b25379786c"
server_key_version: "encryptionkey@0"
I20260621 05:14:47.728930 25476 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260621 05:14:47.729907 25491 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 05:14:47.730175 25476 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260621 05:14:47.730294 25476 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1782018616648578-3598-0/minicluster-data/master-0/data,/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1782018616648578-3598-0/minicluster-data/master-0/wal
uuid: "08a1e3b345fe4c12b2653730a6f2d4c3"
format_stamp: "Formatted at 2026-06-21 05:14:47 on dist-test-slave-vrq1"
server_key: "60fb2f17f5bb7f1293df95613b48ab7b"
server_key_iv: "815221c44d413f09dc1ac6b25379786c"
server_key_version: "encryptionkey@0"
I20260621 05:14:47.730381 25476 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1782018616648578-3598-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1782018616648578-3598-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1782018616648578-3598-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 05:14:47.746081 25476 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 05:14:47.748642 25476 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 05:14:47.748806 25476 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 05:14:47.753551 25476 rpc_server.cc:307] RPC server started. Bound to: 127.3.131.190:32973
I20260621 05:14:47.753643 25543 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.131.190:32973 every 8 connection(s)
I20260621 05:14:47.754011 25476 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1782018616648578-3598-0/minicluster-data/master-0/data/info.pb
I20260621 05:14:47.754944 25544 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 05:14:47.757505 25544 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 08a1e3b345fe4c12b2653730a6f2d4c3: Bootstrap starting.
I20260621 05:14:47.758308 25544 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 08a1e3b345fe4c12b2653730a6f2d4c3: Neither blocks nor log segments found. Creating new log.
I20260621 05:14:47.758613 25544 log.cc:824] T 00000000000000000000000000000000 P 08a1e3b345fe4c12b2653730a6f2d4c3: Log is configured to *not* fsync() on all Append() calls
I20260621 05:14:47.758870 3598 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWZOx2_/build/release/bin/kudu as pid 25476
I20260621 05:14:47.758986 3598 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1782018616648578-3598-0/minicluster-data/master-0/wal/instance
I20260621 05:14:47.759162 3598 external_mini_cluster.cc:1428] Setting key 4ad1053ddf915538b9f5bf4b11628151
I20260621 05:14:47.759613 25544 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 08a1e3b345fe4c12b2653730a6f2d4c3: No bootstrap required, opened a new log
I20260621 05:14:47.761440 25544 raft_consensus.cc:348] T 00000000000000000000000000000000 P 08a1e3b345fe4c12b2653730a6f2d4c3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "08a1e3b345fe4c12b2653730a6f2d4c3" member_type: VOTER last_known_addr { host: "127.3.131.190" port: 32973 } }
I20260621 05:14:47.761617 25544 raft_consensus.cc:374] T 00000000000000000000000000000000 P 08a1e3b345fe4c12b2653730a6f2d4c3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 05:14:47.761682 25544 raft_consensus.cc:729] T 00000000000000000000000000000000 P 08a1e3b345fe4c12b2653730a6f2d4c3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 08a1e3b345fe4c12b2653730a6f2d4c3, State: Initialized, Role: FOLLOWER
I20260621 05:14:47.761830 25544 consensus_queue.cc:260] T 00000000000000000000000000000000 P 08a1e3b345fe4c12b2653730a6f2d4c3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "08a1e3b345fe4c12b2653730a6f2d4c3" member_type: VOTER last_known_addr { host: "127.3.131.190" port: 32973 } }
I20260621 05:14:47.761934 25544 raft_consensus.cc:388] T 00000000000000000000000000000000 P 08a1e3b345fe4c12b2653730a6f2d4c3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260621 05:14:47.761973 25544 raft_consensus.cc:482] T 00000000000000000000000000000000 P 08a1e3b345fe4c12b2653730a6f2d4c3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260621 05:14:47.762035 25544 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 08a1e3b345fe4c12b2653730a6f2d4c3 [term 0 FOLLOWER]: Advancing to term 1
I20260621 05:14:47.762954 25544 raft_consensus.cc:504] T 00000000000000000000000000000000 P 08a1e3b345fe4c12b2653730a6f2d4c3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "08a1e3b345fe4c12b2653730a6f2d4c3" member_type: VOTER last_known_addr { host: "127.3.131.190" port: 32973 } }
I20260621 05:14:47.763095 25544 leader_election.cc:302] T 00000000000000000000000000000000 P 08a1e3b345fe4c12b2653730a6f2d4c3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 08a1e3b345fe4c12b2653730a6f2d4c3; no voters:
I20260621 05:14:47.763327 25544 leader_election.cc:288] T 00000000000000000000000000000000 P 08a1e3b345fe4c12b2653730a6f2d4c3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260621 05:14:47.763361 25549 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 08a1e3b345fe4c12b2653730a6f2d4c3 [term 1 FOLLOWER]: Leader election won for term 1
I20260621 05:14:47.763468 25549 raft_consensus.cc:686] T 00000000000000000000000000000000 P 08a1e3b345fe4c12b2653730a6f2d4c3 [term 1 LEADER]: Becoming Leader. State: Replica: 08a1e3b345fe4c12b2653730a6f2d4c3, State: Running, Role: LEADER
I20260621 05:14:47.763607 25544 sys_catalog.cc:564] T 00000000000000000000000000000000 P 08a1e3b345fe4c12b2653730a6f2d4c3 [sys.catalog]: configured and running, proceeding with master startup.
I20260621 05:14:47.763620 25549 consensus_queue.cc:237] T 00000000000000000000000000000000 P 08a1e3b345fe4c12b2653730a6f2d4c3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "08a1e3b345fe4c12b2653730a6f2d4c3" member_type: VOTER last_known_addr { host: "127.3.131.190" port: 32973 } }
I20260621 05:14:47.764060 25549 sys_catalog.cc:455] T 00000000000000000000000000000000 P 08a1e3b345fe4c12b2653730a6f2d4c3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 08a1e3b345fe4c12b2653730a6f2d4c3. Latest consensus state: current_term: 1 leader_uuid: "08a1e3b345fe4c12b2653730a6f2d4c3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "08a1e3b345fe4c12b2653730a6f2d4c3" member_type: VOTER last_known_addr { host: "127.3.131.190" port: 32973 } } }
I20260621 05:14:47.764213 25549 sys_catalog.cc:458] T 00000000000000000000000000000000 P 08a1e3b345fe4c12b2653730a6f2d4c3 [sys.catalog]: This master's current role is: LEADER
I20260621 05:14:47.764959 25556 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260621 05:14:47.764074 25550 sys_catalog.cc:455] T 00000000000000000000000000000000 P 08a1e3b345fe4c12b2653730a6f2d4c3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "08a1e3b345fe4c12b2653730a6f2d4c3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "08a1e3b345fe4c12b2653730a6f2d4c3" member_type: VOTER last_known_addr { host: "127.3.131.190" port: 32973 } } }
I20260621 05:14:47.765149 25550 sys_catalog.cc:458] T 00000000000000000000000000000000 P 08a1e3b345fe4c12b2653730a6f2d4c3 [sys.catalog]: This master's current role is: LEADER
I20260621 05:14:47.765606 25556 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260621 05:14:47.767784 25556 catalog_manager.cc:1347] Generated new cluster ID: 29d18661fd5c4275a4fd5152493d68f0
I20260621 05:14:47.767860 25556 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Jun 21 05:14:47 dist-test-slave-vrq1 krb5kdc[25463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782018887, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.131.190@KRBTEST.COM
I20260621 05:14:47.773101 25548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:14:47.759755 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42795 (local address 127.3.131.190:32973)
0621 05:14:47.759940 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0621 05:14:47.759948 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0621 05:14:47.759978 (+ 30us) server_negotiation.cc:407] Connection header received
0621 05:14:47.761449 (+ 1471us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:14:47.761459 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:14:47.761521 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:14:47.761758 (+ 237us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:14:47.762542 (+ 784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:14:47.763956 (+ 1414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:14:47.765329 (+ 1373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:14:47.765942 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:14:47.766065 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:14:47.769094 (+ 3029us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:14:47.769099 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:14:47.769099 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:14:47.769112 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:14:47.771275 (+ 2163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:14:47.771752 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:14:47.771755 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:14:47.771756 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:14:47.771884 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:14:47.772128 (+ 244us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:14:47.772129 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:14:47.772129 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:14:47.772438 (+ 309us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:14:47.772597 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0621 05:14:47.772737 (+ 140us) server_negotiation.cc:299] Negotiation successful
0621 05:14:47.772881 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":130,"thread_start_us":92,"threads_started":1}
I20260621 05:14:47.774549 3598 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260621 05:14:47.780885 25556 catalog_manager.cc:1370] Generated new certificate authority record
I20260621 05:14:47.781581 25556 catalog_manager.cc:1504] Loading token signing keys...
I20260621 05:14:47.784233 25548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:14:47.776390 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47895 (local address 127.3.131.190:32973)
0621 05:14:47.776443 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0621 05:14:47.776445 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:14:47.776653 (+ 208us) server_negotiation.cc:407] Connection header received
0621 05:14:47.776722 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:14:47.776722 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:14:47.776733 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:14:47.776781 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:14:47.777490 (+ 709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:14:47.778226 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:14:47.779377 (+ 1151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:14:47.779697 (+ 320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:14:47.779764 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:14:47.781294 (+ 1530us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:14:47.781296 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:14:47.781296 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:14:47.781304 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:14:47.782948 (+ 1644us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:14:47.783421 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:14:47.783422 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:14:47.783423 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:14:47.783482 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:14:47.783700 (+ 218us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:14:47.783701 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:14:47.783701 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:14:47.783822 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:14:47.783923 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0621 05:14:47.784048 (+ 125us) server_negotiation.cc:299] Negotiation successful
0621 05:14:47.784140 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260621 05:14:47.786651 25556 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 08a1e3b345fe4c12b2653730a6f2d4c3: Generated new TSK 0
I20260621 05:14:47.786873 25556 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260621 05:14:47.790576 3598 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWZOx2_/build/release/bin/kudu with pid 25476
2026-06-21T05:14:47Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (1373 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1782018616648578-3598-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 21 05:14:47 dist-test-slave-vrq1 krb5kdc[25575](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 21 05:14:47 dist-test-slave-vrq1 krb5kdc[25575](info): set up 2 sockets
Jun 21 05:14:47 dist-test-slave-vrq1 krb5kdc[25575](info): commencing operation
krb5kdc: starting...
W20260621 05:14:48.855633 3598 mini_kdc.cc:121] Time spent starting KDC: real 1.047s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1782018616648578-3598-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1782018616648578-3598-0/krb5kdc/test-user.keytab.
Jun 21 05:14:48 dist-test-slave-vrq1 krb5kdc[25575](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782018888, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-21T05:14:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-21T05:14:48Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.131.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.131.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.131.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1782018616648578-3598-0/krb5kdc/kudu_127.3.131.190.keytab.
Entry for principal kudu/127.3.131.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1782018616648578-3598-0/krb5kdc/kudu_127.3.131.190.keytab.
I20260621 05:14:48.985746 3598 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWZOx2_/build/release/bin/kudu
/tmp/dist-test-taskWZOx2_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1782018616648578-3598-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1782018616648578-3598-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1782018616648578-3598-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1782018616648578-3598-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.131.190:41019
--webserver_interface=127.3.131.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1782018616648578-3598-0/krb5kdc/kudu_127.3.131.190.keytab
--principal=kudu/127.3.131.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.131.148:43375
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.131.190:41019
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1782018616648578-3598-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1782018616648578-3598-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1782018616648578-3598-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 05:14:49.071581 25588 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 05:14:49.071828 25588 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 05:14:49.071856 25588 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 05:14:49.073225 25588 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 05:14:49.073287 25588 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 05:14:49.073302 25588 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 05:14:49.073316 25588 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 05:14:49.073329 25588 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 05:14:49.075063 25588 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.131.148:43375
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1782018616648578-3598-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1782018616648578-3598-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.131.190:41019
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1782018616648578-3598-0/krb5kdc/kudu_127.3.131.190.keytab
--principal=kudu/127.3.131.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.131.190:41019
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1782018616648578-3598-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.131.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25588
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1782018616648578-3598-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 21 Jun 2026 04:16:16 UTC on ccycloud.quasar-xycaha.root.comops.site
build id 4457881
I20260621 05:14:49.075405 25588 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 05:14:49.075623 25588 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 05:14:49.080883 25593 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:14:49.080888 25594 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:14:49.081107 25588 server_base.cc:1034] running on GCE node
W20260621 05:14:49.080868 25596 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:14:49.081475 25588 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 05:14:49.081835 25588 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 05:14:49.083009 25588 hybrid_clock.cc:648] HybridClock initialized: now 1782018889082985 us; error 36 us; skew 500 ppm
Jun 21 05:14:49 dist-test-slave-vrq1 krb5kdc[25575](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782018889, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.131.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 05:14:49.090929 25588 init.cc:377] Logged in from keytab as kudu/127.3.131.190@KRBTEST.COM (short username kudu)
I20260621 05:14:49.091545 25588 webserver.cc:460] Webserver started at http://127.3.131.190:32917/ using document root <none> and password file <none>
I20260621 05:14:49.091794 25588 fs_manager.cc:359] Metadata directory not provided
I20260621 05:14:49.091843 25588 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 05:14:49.091984 25588 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 05:14:49.093147 25588 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1782018616648578-3598-0/minicluster-data/master-0/data/instance:
uuid: "20224e12677b463a8cb6c1e571921cb0"
format_stamp: "Formatted at 2026-06-21 05:14:49 on dist-test-slave-vrq1"
server_key: "dd5c6b4884d631fdb78a7b1f8553032b"
server_key_iv: "0290241324cfe226e2d163e5e1b4cc54"
server_key_version: "encryptionkey@0"
I20260621 05:14:49.093532 25588 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1782018616648578-3598-0/minicluster-data/master-0/wal/instance:
uuid: "20224e12677b463a8cb6c1e571921cb0"
format_stamp: "Formatted at 2026-06-21 05:14:49 on dist-test-slave-vrq1"
server_key: "dd5c6b4884d631fdb78a7b1f8553032b"
server_key_iv: "0290241324cfe226e2d163e5e1b4cc54"
server_key_version: "encryptionkey@0"
I20260621 05:14:49.095075 25588 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260621 05:14:49.096077 25603 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 05:14:49.096382 25588 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260621 05:14:49.096473 25588 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1782018616648578-3598-0/minicluster-data/master-0/data,/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1782018616648578-3598-0/minicluster-data/master-0/wal
uuid: "20224e12677b463a8cb6c1e571921cb0"
format_stamp: "Formatted at 2026-06-21 05:14:49 on dist-test-slave-vrq1"
server_key: "dd5c6b4884d631fdb78a7b1f8553032b"
server_key_iv: "0290241324cfe226e2d163e5e1b4cc54"
server_key_version: "encryptionkey@0"
I20260621 05:14:49.096572 25588 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1782018616648578-3598-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1782018616648578-3598-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1782018616648578-3598-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 05:14:49.107831 25588 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 05:14:49.110790 25588 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 05:14:49.110967 25588 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 05:14:49.115373 25588 rpc_server.cc:307] RPC server started. Bound to: 127.3.131.190:41019
I20260621 05:14:49.115445 25655 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.131.190:41019 every 8 connection(s)
I20260621 05:14:49.115813 25588 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1782018616648578-3598-0/minicluster-data/master-0/data/info.pb
I20260621 05:14:49.116719 25656 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 05:14:49.119237 25656 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 20224e12677b463a8cb6c1e571921cb0: Bootstrap starting.
I20260621 05:14:49.119889 25656 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 20224e12677b463a8cb6c1e571921cb0: Neither blocks nor log segments found. Creating new log.
I20260621 05:14:49.120201 25656 log.cc:824] T 00000000000000000000000000000000 P 20224e12677b463a8cb6c1e571921cb0: Log is configured to *not* fsync() on all Append() calls
I20260621 05:14:49.121007 25656 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 20224e12677b463a8cb6c1e571921cb0: No bootstrap required, opened a new log
I20260621 05:14:49.121666 3598 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWZOx2_/build/release/bin/kudu as pid 25588
I20260621 05:14:49.121903 3598 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1782018616648578-3598-0/minicluster-data/master-0/wal/instance
I20260621 05:14:49.122073 3598 external_mini_cluster.cc:1428] Setting key f7764162aefc1bd79da05135af792901
I20260621 05:14:49.123466 25656 raft_consensus.cc:348] T 00000000000000000000000000000000 P 20224e12677b463a8cb6c1e571921cb0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "20224e12677b463a8cb6c1e571921cb0" member_type: VOTER last_known_addr { host: "127.3.131.190" port: 41019 } }
I20260621 05:14:49.123660 25656 raft_consensus.cc:374] T 00000000000000000000000000000000 P 20224e12677b463a8cb6c1e571921cb0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 05:14:49.123700 25656 raft_consensus.cc:729] T 00000000000000000000000000000000 P 20224e12677b463a8cb6c1e571921cb0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 20224e12677b463a8cb6c1e571921cb0, State: Initialized, Role: FOLLOWER
I20260621 05:14:49.123874 25656 consensus_queue.cc:260] T 00000000000000000000000000000000 P 20224e12677b463a8cb6c1e571921cb0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "20224e12677b463a8cb6c1e571921cb0" member_type: VOTER last_known_addr { host: "127.3.131.190" port: 41019 } }
I20260621 05:14:49.123968 25656 raft_consensus.cc:388] T 00000000000000000000000000000000 P 20224e12677b463a8cb6c1e571921cb0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260621 05:14:49.124001 25656 raft_consensus.cc:482] T 00000000000000000000000000000000 P 20224e12677b463a8cb6c1e571921cb0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260621 05:14:49.124045 25656 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 20224e12677b463a8cb6c1e571921cb0 [term 0 FOLLOWER]: Advancing to term 1
I20260621 05:14:49.124933 25656 raft_consensus.cc:504] T 00000000000000000000000000000000 P 20224e12677b463a8cb6c1e571921cb0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "20224e12677b463a8cb6c1e571921cb0" member_type: VOTER last_known_addr { host: "127.3.131.190" port: 41019 } }
I20260621 05:14:49.125049 25656 leader_election.cc:302] T 00000000000000000000000000000000 P 20224e12677b463a8cb6c1e571921cb0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 20224e12677b463a8cb6c1e571921cb0; no voters:
I20260621 05:14:49.125224 25656 leader_election.cc:288] T 00000000000000000000000000000000 P 20224e12677b463a8cb6c1e571921cb0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260621 05:14:49.125349 25661 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 20224e12677b463a8cb6c1e571921cb0 [term 1 FOLLOWER]: Leader election won for term 1
I20260621 05:14:49.125519 25656 sys_catalog.cc:564] T 00000000000000000000000000000000 P 20224e12677b463a8cb6c1e571921cb0 [sys.catalog]: configured and running, proceeding with master startup.
I20260621 05:14:49.125607 25661 raft_consensus.cc:686] T 00000000000000000000000000000000 P 20224e12677b463a8cb6c1e571921cb0 [term 1 LEADER]: Becoming Leader. State: Replica: 20224e12677b463a8cb6c1e571921cb0, State: Running, Role: LEADER
I20260621 05:14:49.125736 25661 consensus_queue.cc:237] T 00000000000000000000000000000000 P 20224e12677b463a8cb6c1e571921cb0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "20224e12677b463a8cb6c1e571921cb0" member_type: VOTER last_known_addr { host: "127.3.131.190" port: 41019 } }
I20260621 05:14:49.126120 25661 sys_catalog.cc:455] T 00000000000000000000000000000000 P 20224e12677b463a8cb6c1e571921cb0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 20224e12677b463a8cb6c1e571921cb0. Latest consensus state: current_term: 1 leader_uuid: "20224e12677b463a8cb6c1e571921cb0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "20224e12677b463a8cb6c1e571921cb0" member_type: VOTER last_known_addr { host: "127.3.131.190" port: 41019 } } }
I20260621 05:14:49.126255 25661 sys_catalog.cc:458] T 00000000000000000000000000000000 P 20224e12677b463a8cb6c1e571921cb0 [sys.catalog]: This master's current role is: LEADER
I20260621 05:14:49.126497 25670 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260621 05:14:49.127048 25670 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260621 05:14:49.127981 25662 sys_catalog.cc:455] T 00000000000000000000000000000000 P 20224e12677b463a8cb6c1e571921cb0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "20224e12677b463a8cb6c1e571921cb0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "20224e12677b463a8cb6c1e571921cb0" member_type: VOTER last_known_addr { host: "127.3.131.190" port: 41019 } } }
I20260621 05:14:49.128080 25662 sys_catalog.cc:458] T 00000000000000000000000000000000 P 20224e12677b463a8cb6c1e571921cb0 [sys.catalog]: This master's current role is: LEADER
I20260621 05:14:49.129226 25670 catalog_manager.cc:1347] Generated new cluster ID: 9bc7f3eec125447789cedc81c0e27217
I20260621 05:14:49.129251 25660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:14:49.122709 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57489 (local address 127.3.131.190:41019)
0621 05:14:49.122952 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0621 05:14:49.122963 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0621 05:14:49.123004 (+ 41us) server_negotiation.cc:407] Connection header received
0621 05:14:49.123414 (+ 410us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:14:49.123424 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:14:49.123495 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:14:49.123669 (+ 174us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:14:49.123938 (+ 269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:14:49.125695 (+ 1757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:14:49.126888 (+ 1193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:14:49.127549 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:14:49.127670 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:14:49.128412 (+ 742us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:14:49.128416 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:14:49.128416 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 05:14:49.128664 (+ 248us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:14:49.128709 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 05:14:49.128727 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:14:49.128795 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0621 05:14:49.128922 (+ 127us) server_negotiation.cc:299] Negotiation successful
0621 05:14:49.128990 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":171,"thread_start_us":97,"threads_started":1}
I20260621 05:14:49.129297 25670 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260621 05:14:49.130198 3598 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260621 05:14:49.136059 25660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:14:49.131879 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40517 (local address 127.3.131.190:41019)
0621 05:14:49.131923 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0621 05:14:49.131925 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:14:49.132131 (+ 206us) server_negotiation.cc:407] Connection header received
0621 05:14:49.132197 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:14:49.132198 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:14:49.132209 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:14:49.132273 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:14:49.132421 (+ 148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:14:49.133126 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:14:49.134168 (+ 1042us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:14:49.134458 (+ 290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:14:49.134524 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:14:49.135671 (+ 1147us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:14:49.135673 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:14:49.135673 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 05:14:49.135680 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:14:49.135699 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 05:14:49.135705 (+ 6us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:14:49.135768 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0621 05:14:49.135903 (+ 135us) server_negotiation.cc:299] Negotiation successful
0621 05:14:49.135933 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260621 05:14:49.142153 25670 catalog_manager.cc:1370] Generated new certificate authority record
I20260621 05:14:49.142815 25670 catalog_manager.cc:1504] Loading token signing keys...
I20260621 05:14:49.145965 25670 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 20224e12677b463a8cb6c1e571921cb0: Generated new TSK 0
I20260621 05:14:49.146202 25670 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260621 05:14:49.152571 3598 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWZOx2_/build/release/bin/kudu with pid 25588
2026-06-21T05:14:49Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1362 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1782018616648578-3598-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 21 05:14:49 dist-test-slave-vrq1 krb5kdc[25687](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 21 05:14:49 dist-test-slave-vrq1 krb5kdc[25687](info): set up 2 sockets
Jun 21 05:14:49 dist-test-slave-vrq1 krb5kdc[25687](info): commencing operation
krb5kdc: starting...
W20260621 05:14:50.211638 3598 mini_kdc.cc:121] Time spent starting KDC: real 1.040s user 0.002s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1782018616648578-3598-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1782018616648578-3598-0/krb5kdc/test-user.keytab.
Jun 21 05:14:50 dist-test-slave-vrq1 krb5kdc[25687](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782018890, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-21T05:14:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-21T05:14:50Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.131.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.131.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.131.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1782018616648578-3598-0/krb5kdc/kudu_127.3.131.190.keytab.
Entry for principal kudu/127.3.131.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1782018616648578-3598-0/krb5kdc/kudu_127.3.131.190.keytab.
I20260621 05:14:50.336298 3598 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWZOx2_/build/release/bin/kudu
/tmp/dist-test-taskWZOx2_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1782018616648578-3598-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1782018616648578-3598-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1782018616648578-3598-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1782018616648578-3598-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.131.190:37535
--webserver_interface=127.3.131.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1782018616648578-3598-0/krb5kdc/kudu_127.3.131.190.keytab
--principal=kudu/127.3.131.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.131.148:38557
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.131.190:37535
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1782018616648578-3598-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1782018616648578-3598-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1782018616648578-3598-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 05:14:50.417550 25700 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 05:14:50.417778 25700 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 05:14:50.417800 25700 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 05:14:50.418911 25700 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 05:14:50.418949 25700 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 05:14:50.418963 25700 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 05:14:50.418977 25700 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 05:14:50.418988 25700 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 05:14:50.420459 25700 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.131.148:38557
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1782018616648578-3598-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1782018616648578-3598-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.131.190:37535
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1782018616648578-3598-0/krb5kdc/kudu_127.3.131.190.keytab
--principal=kudu/127.3.131.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.131.190:37535
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1782018616648578-3598-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.131.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25700
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1782018616648578-3598-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 21 Jun 2026 04:16:16 UTC on ccycloud.quasar-xycaha.root.comops.site
build id 4457881
I20260621 05:14:50.420672 25700 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 05:14:50.420922 25700 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 05:14:50.426208 25706 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:14:50.426208 25705 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:14:50.426250 25708 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:14:50.426476 25700 server_base.cc:1034] running on GCE node
I20260621 05:14:50.426823 25700 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 05:14:50.427142 25700 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 05:14:50.428336 25700 hybrid_clock.cc:648] HybridClock initialized: now 1782018890428307 us; error 49 us; skew 500 ppm
Jun 21 05:14:50 dist-test-slave-vrq1 krb5kdc[25687](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782018890, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.131.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 05:14:50.431146 25700 init.cc:377] Logged in from keytab as kudu/127.3.131.190@KRBTEST.COM (short username kudu)
I20260621 05:14:50.431752 25700 webserver.cc:460] Webserver started at http://127.3.131.190:39757/ using document root <none> and password file <none>
I20260621 05:14:50.431998 25700 fs_manager.cc:359] Metadata directory not provided
I20260621 05:14:50.432066 25700 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 05:14:50.432224 25700 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 05:14:50.433339 25700 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1782018616648578-3598-0/minicluster-data/master-0/data/instance:
uuid: "73fd0e8342e847a19c6a92b0bbe72ad4"
format_stamp: "Formatted at 2026-06-21 05:14:50 on dist-test-slave-vrq1"
server_key: "d4d8f8c33bf067c24aca9d606248dd5c"
server_key_iv: "a429b84c72924852e7e43b42c60ee8a0"
server_key_version: "encryptionkey@0"
I20260621 05:14:50.433734 25700 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1782018616648578-3598-0/minicluster-data/master-0/wal/instance:
uuid: "73fd0e8342e847a19c6a92b0bbe72ad4"
format_stamp: "Formatted at 2026-06-21 05:14:50 on dist-test-slave-vrq1"
server_key: "d4d8f8c33bf067c24aca9d606248dd5c"
server_key_iv: "a429b84c72924852e7e43b42c60ee8a0"
server_key_version: "encryptionkey@0"
I20260621 05:14:50.435137 25700 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260621 05:14:50.436087 25715 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 05:14:50.436301 25700 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260621 05:14:50.436405 25700 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1782018616648578-3598-0/minicluster-data/master-0/data,/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1782018616648578-3598-0/minicluster-data/master-0/wal
uuid: "73fd0e8342e847a19c6a92b0bbe72ad4"
format_stamp: "Formatted at 2026-06-21 05:14:50 on dist-test-slave-vrq1"
server_key: "d4d8f8c33bf067c24aca9d606248dd5c"
server_key_iv: "a429b84c72924852e7e43b42c60ee8a0"
server_key_version: "encryptionkey@0"
I20260621 05:14:50.436481 25700 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1782018616648578-3598-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1782018616648578-3598-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1782018616648578-3598-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 05:14:50.439890 25700 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 05:14:50.442027 25700 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 05:14:50.442178 25700 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 05:14:50.446321 25700 rpc_server.cc:307] RPC server started. Bound to: 127.3.131.190:37535
I20260621 05:14:50.446355 25767 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.131.190:37535 every 8 connection(s)
I20260621 05:14:50.446707 25700 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1782018616648578-3598-0/minicluster-data/master-0/data/info.pb
I20260621 05:14:50.447789 25768 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 05:14:50.450028 25768 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 73fd0e8342e847a19c6a92b0bbe72ad4: Bootstrap starting.
I20260621 05:14:50.450692 25768 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 73fd0e8342e847a19c6a92b0bbe72ad4: Neither blocks nor log segments found. Creating new log.
I20260621 05:14:50.450978 25768 log.cc:824] T 00000000000000000000000000000000 P 73fd0e8342e847a19c6a92b0bbe72ad4: Log is configured to *not* fsync() on all Append() calls
I20260621 05:14:50.451371 3598 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWZOx2_/build/release/bin/kudu as pid 25700
I20260621 05:14:50.451499 3598 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1782018616648578-3598-0/minicluster-data/master-0/wal/instance
I20260621 05:14:50.451675 3598 external_mini_cluster.cc:1428] Setting key fef2d2e911da4de860e0b74a4862f776
I20260621 05:14:50.451808 25768 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 73fd0e8342e847a19c6a92b0bbe72ad4: No bootstrap required, opened a new log
I20260621 05:14:50.453606 25768 raft_consensus.cc:348] T 00000000000000000000000000000000 P 73fd0e8342e847a19c6a92b0bbe72ad4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "73fd0e8342e847a19c6a92b0bbe72ad4" member_type: VOTER last_known_addr { host: "127.3.131.190" port: 37535 } }
I20260621 05:14:50.453765 25768 raft_consensus.cc:374] T 00000000000000000000000000000000 P 73fd0e8342e847a19c6a92b0bbe72ad4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 05:14:50.453812 25768 raft_consensus.cc:729] T 00000000000000000000000000000000 P 73fd0e8342e847a19c6a92b0bbe72ad4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73fd0e8342e847a19c6a92b0bbe72ad4, State: Initialized, Role: FOLLOWER
I20260621 05:14:50.453943 25768 consensus_queue.cc:260] T 00000000000000000000000000000000 P 73fd0e8342e847a19c6a92b0bbe72ad4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "73fd0e8342e847a19c6a92b0bbe72ad4" member_type: VOTER last_known_addr { host: "127.3.131.190" port: 37535 } }
I20260621 05:14:50.454035 25768 raft_consensus.cc:388] T 00000000000000000000000000000000 P 73fd0e8342e847a19c6a92b0bbe72ad4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260621 05:14:50.454077 25768 raft_consensus.cc:482] T 00000000000000000000000000000000 P 73fd0e8342e847a19c6a92b0bbe72ad4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260621 05:14:50.454115 25768 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 73fd0e8342e847a19c6a92b0bbe72ad4 [term 0 FOLLOWER]: Advancing to term 1
I20260621 05:14:50.455013 25772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:14:50.452197 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48981 (local address 127.3.131.190:37535)
0621 05:14:50.452406 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0621 05:14:50.452416 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0621 05:14:50.452452 (+ 36us) server_negotiation.cc:407] Connection header received
0621 05:14:50.453602 (+ 1150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:14:50.453610 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:14:50.453641 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:14:50.453844 (+ 203us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:14:50.454572 (+ 728us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:14:50.454575 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:14:50.454576 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 05:14:50.454580 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:14:50.454631 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 05:14:50.454643 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:14:50.454724 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0621 05:14:50.454810 (+ 86us) server_negotiation.cc:299] Negotiation successful
0621 05:14:50.454852 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":137,"thread_start_us":75,"threads_started":1}
I20260621 05:14:50.455051 25768 raft_consensus.cc:504] T 00000000000000000000000000000000 P 73fd0e8342e847a19c6a92b0bbe72ad4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "73fd0e8342e847a19c6a92b0bbe72ad4" member_type: VOTER last_known_addr { host: "127.3.131.190" port: 37535 } }
I20260621 05:14:50.455168 25768 leader_election.cc:302] T 00000000000000000000000000000000 P 73fd0e8342e847a19c6a92b0bbe72ad4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 73fd0e8342e847a19c6a92b0bbe72ad4; no voters:
I20260621 05:14:50.455387 25768 leader_election.cc:288] T 00000000000000000000000000000000 P 73fd0e8342e847a19c6a92b0bbe72ad4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260621 05:14:50.455440 25773 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 73fd0e8342e847a19c6a92b0bbe72ad4 [term 1 FOLLOWER]: Leader election won for term 1
I20260621 05:14:50.455637 25773 raft_consensus.cc:686] T 00000000000000000000000000000000 P 73fd0e8342e847a19c6a92b0bbe72ad4 [term 1 LEADER]: Becoming Leader. State: Replica: 73fd0e8342e847a19c6a92b0bbe72ad4, State: Running, Role: LEADER
I20260621 05:14:50.455664 25768 sys_catalog.cc:564] T 00000000000000000000000000000000 P 73fd0e8342e847a19c6a92b0bbe72ad4 [sys.catalog]: configured and running, proceeding with master startup.
I20260621 05:14:50.455781 25773 consensus_queue.cc:237] T 00000000000000000000000000000000 P 73fd0e8342e847a19c6a92b0bbe72ad4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "73fd0e8342e847a19c6a92b0bbe72ad4" member_type: VOTER last_known_addr { host: "127.3.131.190" port: 37535 } }
I20260621 05:14:50.456152 3598 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260621 05:14:50.456238 25775 sys_catalog.cc:455] T 00000000000000000000000000000000 P 73fd0e8342e847a19c6a92b0bbe72ad4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 73fd0e8342e847a19c6a92b0bbe72ad4. Latest consensus state: current_term: 1 leader_uuid: "73fd0e8342e847a19c6a92b0bbe72ad4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "73fd0e8342e847a19c6a92b0bbe72ad4" member_type: VOTER last_known_addr { host: "127.3.131.190" port: 37535 } } }
I20260621 05:14:50.456429 25775 sys_catalog.cc:458] T 00000000000000000000000000000000 P 73fd0e8342e847a19c6a92b0bbe72ad4 [sys.catalog]: This master's current role is: LEADER
I20260621 05:14:50.456370 25774 sys_catalog.cc:455] T 00000000000000000000000000000000 P 73fd0e8342e847a19c6a92b0bbe72ad4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "73fd0e8342e847a19c6a92b0bbe72ad4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "73fd0e8342e847a19c6a92b0bbe72ad4" member_type: VOTER last_known_addr { host: "127.3.131.190" port: 37535 } } }
I20260621 05:14:50.456624 25782 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260621 05:14:50.456634 25774 sys_catalog.cc:458] T 00000000000000000000000000000000 P 73fd0e8342e847a19c6a92b0bbe72ad4 [sys.catalog]: This master's current role is: LEADER
I20260621 05:14:50.457199 25782 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260621 05:14:50.459007 25772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:14:50.457799 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49167 (local address 127.3.131.190:37535)
0621 05:14:50.457838 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0621 05:14:50.457841 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0621 05:14:50.457959 (+ 118us) server_negotiation.cc:407] Connection header received
0621 05:14:50.458020 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:14:50.458021 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:14:50.458034 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:14:50.458094 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:14:50.458773 (+ 679us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:14:50.458775 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:14:50.458775 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 05:14:50.458782 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:14:50.458797 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 05:14:50.458803 (+ 6us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:14:50.458867 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0621 05:14:50.458928 (+ 61us) server_negotiation.cc:299] Negotiation successful
0621 05:14:50.458943 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260621 05:14:50.459146 25782 catalog_manager.cc:1347] Generated new cluster ID: 10f86df69dbb4a40969006e864b998f5
I20260621 05:14:50.459218 25782 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260621 05:14:50.470188 25782 catalog_manager.cc:1370] Generated new certificate authority record
I20260621 05:14:50.470398 25782 catalog_manager.cc:1504] Loading token signing keys...
I20260621 05:14:50.489566 25782 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 73fd0e8342e847a19c6a92b0bbe72ad4: Generated new TSK 0
I20260621 05:14:50.489867 25782 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260621 05:14:50.493623 3598 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWZOx2_/build/release/bin/kudu with pid 25700
2026-06-21T05:14:50Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1341 ms)
[----------] 18 tests from AuthTokenIssuingTest (24707 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782018616648578-3598-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 21 05:14:50 dist-test-slave-vrq1 krb5kdc[25799](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 21 05:14:50 dist-test-slave-vrq1 krb5kdc[25799](info): set up 2 sockets
Jun 21 05:14:50 dist-test-slave-vrq1 krb5kdc[25799](info): commencing operation
krb5kdc: starting...
W20260621 05:14:51.567716 3598 mini_kdc.cc:121] Time spent starting KDC: real 1.055s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782018616648578-3598-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782018616648578-3598-0/krb5kdc/test-user.keytab.
Jun 21 05:14:51 dist-test-slave-vrq1 krb5kdc[25799](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782018891, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-21T05:14:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-21T05:14:51Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.131.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.131.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.131.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782018616648578-3598-0/krb5kdc/kudu_127.3.131.190.keytab.
Entry for principal kudu/127.3.131.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782018616648578-3598-0/krb5kdc/kudu_127.3.131.190.keytab.
I20260621 05:14:51.694823 3598 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWZOx2_/build/release/bin/kudu
/tmp/dist-test-taskWZOx2_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782018616648578-3598-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782018616648578-3598-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782018616648578-3598-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782018616648578-3598-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.131.190:39053
--webserver_interface=127.3.131.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782018616648578-3598-0/krb5kdc/kudu_127.3.131.190.keytab
--principal=kudu/127.3.131.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.131.148:44517
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.131.190:39053,127.3.131.189:42239,127.3.131.188:35325
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782018616648578-3598-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782018616648578-3598-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782018616648578-3598-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 05:14:51.779520 25812 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260621 05:14:51.779827 25812 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 05:14:51.779862 25812 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 05:14:51.779913 25812 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 05:14:51.781198 25812 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 05:14:51.781275 25812 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 05:14:51.781303 25812 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 05:14:51.781327 25812 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 05:14:51.781359 25812 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 05:14:51.783048 25812 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.131.148:44517
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782018616648578-3598-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782018616648578-3598-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.131.190:39053,127.3.131.189:42239,127.3.131.188:35325
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782018616648578-3598-0/krb5kdc/kudu_127.3.131.190.keytab
--principal=kudu/127.3.131.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.131.190:39053
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782018616648578-3598-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.131.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25812
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782018616648578-3598-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 21 Jun 2026 04:16:16 UTC on ccycloud.quasar-xycaha.root.comops.site
build id 4457881
I20260621 05:14:51.783391 25812 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 05:14:51.783660 25812 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 05:14:51.789310 25818 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:14:51.789294 25820 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:14:51.789294 25817 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:14:51.789634 25812 server_base.cc:1034] running on GCE node
I20260621 05:14:51.789880 25812 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 05:14:51.790197 25812 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 05:14:51.791380 25812 hybrid_clock.cc:648] HybridClock initialized: now 1782018891791354 us; error 35 us; skew 500 ppm
Jun 21 05:14:51 dist-test-slave-vrq1 krb5kdc[25799](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782018891, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.131.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 05:14:51.794166 25812 init.cc:377] Logged in from keytab as kudu/127.3.131.190@KRBTEST.COM (short username kudu)
I20260621 05:14:51.794651 25812 webserver.cc:460] Webserver started at http://127.3.131.190:44637/ using document root <none> and password file <none>
I20260621 05:14:51.794888 25812 fs_manager.cc:359] Metadata directory not provided
I20260621 05:14:51.794945 25812 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 05:14:51.795064 25812 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 05:14:51.796254 25812 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782018616648578-3598-0/minicluster-data/master-0/data/instance:
uuid: "f7023899fb034854a56325cab379fabe"
format_stamp: "Formatted at 2026-06-21 05:14:51 on dist-test-slave-vrq1"
server_key: "a73090e0b049e6fac63afd77aa83c08c"
server_key_iv: "233e7cd6bd177adf9fc2104c6b7f27a6"
server_key_version: "encryptionkey@0"
I20260621 05:14:51.796626 25812 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782018616648578-3598-0/minicluster-data/master-0/wal/instance:
uuid: "f7023899fb034854a56325cab379fabe"
format_stamp: "Formatted at 2026-06-21 05:14:51 on dist-test-slave-vrq1"
server_key: "a73090e0b049e6fac63afd77aa83c08c"
server_key_iv: "233e7cd6bd177adf9fc2104c6b7f27a6"
server_key_version: "encryptionkey@0"
I20260621 05:14:51.798064 25812 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260621 05:14:51.798884 25827 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 05:14:51.799149 25812 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260621 05:14:51.799296 25812 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782018616648578-3598-0/minicluster-data/master-0/data,/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782018616648578-3598-0/minicluster-data/master-0/wal
uuid: "f7023899fb034854a56325cab379fabe"
format_stamp: "Formatted at 2026-06-21 05:14:51 on dist-test-slave-vrq1"
server_key: "a73090e0b049e6fac63afd77aa83c08c"
server_key_iv: "233e7cd6bd177adf9fc2104c6b7f27a6"
server_key_version: "encryptionkey@0"
I20260621 05:14:51.799373 25812 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782018616648578-3598-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782018616648578-3598-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782018616648578-3598-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 05:14:51.820278 25812 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 05:14:51.822834 25812 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 05:14:51.822976 25812 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 05:14:51.827363 25812 rpc_server.cc:307] RPC server started. Bound to: 127.3.131.190:39053
I20260621 05:14:51.827430 25879 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.131.190:39053 every 8 connection(s)
I20260621 05:14:51.827782 25812 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782018616648578-3598-0/minicluster-data/master-0/data/info.pb
I20260621 05:14:51.828840 25880 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 05:14:51.830503 3598 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWZOx2_/build/release/bin/kudu as pid 25812
I20260621 05:14:51.830614 3598 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782018616648578-3598-0/minicluster-data/master-0/wal/instance
I20260621 05:14:51.830807 3598 external_mini_cluster.cc:1428] Setting key 8d1abaca9a63ccd0ec10d75d80a9eaa6
I20260621 05:14:51.830605 25880 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.131.190" port: 39053 } has no permanent_uuid. Determining permanent_uuid...
Jun 21 05:14:51 dist-test-slave-vrq1 krb5kdc[25799](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782018891, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.131.190@KRBTEST.COM for kudu/127.3.131.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.3.131.189@KRBTEST.COM; defaulting to no policy
I20260621 05:14:51.841961 25883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:14:51.831618 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.131.190:39053 (local address 127.0.0.1:38650)
0621 05:14:51.831839 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0621 05:14:51.831879 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0621 05:14:51.832583 (+ 704us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:14:51.832983 (+ 400us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:14:51.832986 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:14:51.833255 (+ 269us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:14:51.833431 (+ 176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:14:51.833435 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:14:51.834706 (+ 1271us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:14:51.834707 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:14:51.835538 (+ 831us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:14:51.835540 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:14:51.836066 (+ 526us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:14:51.836067 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:14:51.836130 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:14:51.836709 (+ 579us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 05:14:51.836718 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:14:51.838366 (+ 1648us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 05:14:51.840338 (+ 1972us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:14:51.840345 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:14:51.840347 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:14:51.840552 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:14:51.840824 (+ 272us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:14:51.840824 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:14:51.840825 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:14:51.840925 (+ 100us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:14:51.841311 (+ 386us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 05:14:51.841317 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 05:14:51.841381 (+ 64us) client_negotiation.cc:769] Sending connection context
0621 05:14:51.841490 (+ 109us) client_negotiation.cc:240] Negotiation successful
0621 05:14:51.841749 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":178,"mutex_wait_us":115,"thread_start_us":134,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.131.189@KRBTEST.COM" created.
I20260621 05:14:51.841972 25882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:14:51.831651 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38650 (local address 127.3.131.190:39053)
0621 05:14:51.831825 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0621 05:14:51.831835 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0621 05:14:51.832066 (+ 231us) server_negotiation.cc:407] Connection header received
0621 05:14:51.832802 (+ 736us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:14:51.832804 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:14:51.832829 (+ 25us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:14:51.832942 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:14:51.833601 (+ 659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:14:51.834562 (+ 961us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:14:51.835632 (+ 1070us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:14:51.835950 (+ 318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:14:51.836084 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:14:51.838472 (+ 2388us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:14:51.838480 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:14:51.838480 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:14:51.838487 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:14:51.840229 (+ 1742us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:14:51.840657 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:14:51.840661 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:14:51.840661 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:14:51.840751 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:14:51.840991 (+ 240us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:14:51.840992 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:14:51.840993 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:14:51.841232 (+ 239us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:14:51.841321 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0621 05:14:51.841522 (+ 201us) server_negotiation.cc:299] Negotiation successful
0621 05:14:51.841700 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":7,"server-negotiator.queue_time_us":132,"thread_start_us":84,"threads_started":1}
I20260621 05:14:51.843197 25880 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.131.189" port: 42239 } has no permanent_uuid. Determining permanent_uuid...
I20260621 05:14:51.843619 25883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:14:51.843473 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.131.189:42239 (local address 127.0.0.1:49520)
0621 05:14:51.843543 (+ 70us) negotiation.cc:107] Waiting for socket to connect
0621 05:14:51.843591 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.131.189:42239: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260621 05:14:51.843804 25831 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.131.189:42239: connect: Connection refused (error 111)
I20260621 05:14:51.844324 25883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:14:51.844189 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.131.189:42239 (local address 127.0.0.1:49526)
0621 05:14:51.844269 (+ 80us) negotiation.cc:107] Waiting for socket to connect
0621 05:14:51.844300 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.131.189:42239: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
W20260621 05:14:51.844492 25880 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.131.189:42239: Network error: Client connection negotiation failed: client connection to 127.3.131.189:42239: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.131.189 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782018616648578-3598-0/krb5kdc/kudu_127.3.131.189.keytab.
Entry for principal kudu/127.3.131.189 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782018616648578-3598-0/krb5kdc/kudu_127.3.131.189.keytab.
I20260621 05:14:51.868144 3598 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWZOx2_/build/release/bin/kudu
/tmp/dist-test-taskWZOx2_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782018616648578-3598-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782018616648578-3598-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782018616648578-3598-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782018616648578-3598-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.131.189:42239
--webserver_interface=127.3.131.189
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782018616648578-3598-0/krb5kdc/kudu_127.3.131.189.keytab
--principal=kudu/127.3.131.189
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.131.148:44517
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.131.190:39053,127.3.131.189:42239,127.3.131.188:35325
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782018616648578-3598-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782018616648578-3598-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782018616648578-3598-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260621 05:14:51.893761 25880 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.131.189" port: 42239 } attempt: 1
I20260621 05:14:51.894371 25883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:14:51.894207 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.131.189:42239 (local address 127.0.0.1:49540)
0621 05:14:51.894294 (+ 87us) negotiation.cc:107] Waiting for socket to connect
0621 05:14:51.894338 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.131.189:42239: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
I20260621 05:14:51.894805 25883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:14:51.894682 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.131.189:42239 (local address 127.0.0.1:49548)
0621 05:14:51.894753 (+ 71us) negotiation.cc:107] Waiting for socket to connect
0621 05:14:51.894783 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.131.189:42239: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260621 05:14:51.894937 25880 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.131.189:42239: Network error: Client connection negotiation failed: client connection to 127.3.131.189:42239: connect: Connection refused (error 111)
W20260621 05:14:51.952430 25887 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260621 05:14:51.952682 25887 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 05:14:51.952701 25887 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 05:14:51.952724 25887 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 05:14:51.953981 25887 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 05:14:51.954032 25887 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 05:14:51.954047 25887 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 05:14:51.954061 25887 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 05:14:51.954074 25887 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 05:14:51.955704 25887 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.131.148:44517
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782018616648578-3598-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782018616648578-3598-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.3.131.190:39053,127.3.131.189:42239,127.3.131.188:35325
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782018616648578-3598-0/krb5kdc/kudu_127.3.131.189.keytab
--principal=kudu/127.3.131.189
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.131.189:42239
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782018616648578-3598-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.3.131.189
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25887
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782018616648578-3598-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 21 Jun 2026 04:16:16 UTC on ccycloud.quasar-xycaha.root.comops.site
build id 4457881
I20260621 05:14:51.955919 25887 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 05:14:51.956100 25887 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 05:14:51.961180 25892 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:14:51.961230 25887 server_base.cc:1034] running on GCE node
W20260621 05:14:51.961227 25895 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:14:51.961191 25893 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:14:51.961720 25887 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 05:14:51.962075 25887 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 05:14:51.963071 25880 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.131.189" port: 42239 } attempt: 2
I20260621 05:14:51.963277 25887 hybrid_clock.cc:648] HybridClock initialized: now 1782018891963249 us; error 40 us; skew 500 ppm
I20260621 05:14:51.963538 25883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:14:51.963369 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.131.189:42239 (local address 127.0.0.1:49554)
0621 05:14:51.963441 (+ 72us) negotiation.cc:107] Waiting for socket to connect
0621 05:14:51.963490 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.131.189:42239: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
I20260621 05:14:51.963956 25883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:14:51.963860 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.131.189:42239 (local address 127.0.0.1:49568)
0621 05:14:51.963908 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0621 05:14:51.963934 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.131.189:42239: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260621 05:14:51.964084 25880 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.131.189:42239: Network error: Client connection negotiation failed: client connection to 127.3.131.189:42239: connect: Connection refused (error 111)
Jun 21 05:14:51 dist-test-slave-vrq1 krb5kdc[25799](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782018891, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.131.189@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 05:14:51.966164 25887 init.cc:377] Logged in from keytab as kudu/127.3.131.189@KRBTEST.COM (short username kudu)
I20260621 05:14:51.966655 25887 webserver.cc:460] Webserver started at http://127.3.131.189:38203/ using document root <none> and password file <none>
I20260621 05:14:51.966900 25887 fs_manager.cc:359] Metadata directory not provided
I20260621 05:14:51.966969 25887 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 05:14:51.967113 25887 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 05:14:51.968214 25887 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782018616648578-3598-0/minicluster-data/master-1/data/instance:
uuid: "c4e36f6e2d6b4e2f9e95bdf4b99f2d08"
format_stamp: "Formatted at 2026-06-21 05:14:51 on dist-test-slave-vrq1"
server_key: "020eb395244b53b306555066aa7b799b"
server_key_iv: "368f9350620d4e0164953f512c516988"
server_key_version: "encryptionkey@0"
I20260621 05:14:51.968611 25887 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782018616648578-3598-0/minicluster-data/master-1/wal/instance:
uuid: "c4e36f6e2d6b4e2f9e95bdf4b99f2d08"
format_stamp: "Formatted at 2026-06-21 05:14:51 on dist-test-slave-vrq1"
server_key: "020eb395244b53b306555066aa7b799b"
server_key_iv: "368f9350620d4e0164953f512c516988"
server_key_version: "encryptionkey@0"
I20260621 05:14:51.969986 25887 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260621 05:14:51.970917 25902 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 05:14:51.971202 25887 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260621 05:14:51.971297 25887 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782018616648578-3598-0/minicluster-data/master-1/data,/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782018616648578-3598-0/minicluster-data/master-1/wal
uuid: "c4e36f6e2d6b4e2f9e95bdf4b99f2d08"
format_stamp: "Formatted at 2026-06-21 05:14:51 on dist-test-slave-vrq1"
server_key: "020eb395244b53b306555066aa7b799b"
server_key_iv: "368f9350620d4e0164953f512c516988"
server_key_version: "encryptionkey@0"
I20260621 05:14:51.971391 25887 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782018616648578-3598-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782018616648578-3598-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782018616648578-3598-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 05:14:51.987812 25887 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 05:14:51.990077 25887 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 05:14:51.990262 25887 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 05:14:51.994765 25887 rpc_server.cc:307] RPC server started. Bound to: 127.3.131.189:42239
I20260621 05:14:51.994801 25954 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.131.189:42239 every 8 connection(s)
I20260621 05:14:51.995239 25887 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782018616648578-3598-0/minicluster-data/master-1/data/info.pb
I20260621 05:14:51.996404 25955 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 05:14:51.998173 25955 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.131.190" port: 39053 } has no permanent_uuid. Determining permanent_uuid...
I20260621 05:14:52.003741 3598 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWZOx2_/build/release/bin/kudu as pid 25887
I20260621 05:14:52.003871 3598 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782018616648578-3598-0/minicluster-data/master-1/wal/instance
I20260621 05:14:52.004101 3598 external_mini_cluster.cc:1428] Setting key 282499bf0e6179992c7f7a4c805153b1
Jun 21 05:14:52 dist-test-slave-vrq1 krb5kdc[25799](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782018891, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.131.189@KRBTEST.COM for kudu/127.3.131.190@KRBTEST.COM
I20260621 05:14:52.009722 25882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:14:51.999436 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38652 (local address 127.3.131.190:39053)
0621 05:14:51.999552 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0621 05:14:51.999554 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:14:51.999738 (+ 184us) server_negotiation.cc:407] Connection header received
0621 05:14:52.000319 (+ 581us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:14:52.000321 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:14:52.000338 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:14:52.000394 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:14:52.001079 (+ 685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:14:52.001635 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:14:52.002705 (+ 1070us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:14:52.002923 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:14:52.003005 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:14:52.006100 (+ 3095us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:14:52.006102 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:14:52.006103 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:14:52.006111 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:14:52.008121 (+ 2010us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:14:52.008631 (+ 510us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:14:52.008633 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:14:52.008633 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:14:52.008715 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:14:52.008994 (+ 279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:14:52.008995 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:14:52.008995 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:14:52.009151 (+ 156us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:14:52.009234 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0621 05:14:52.009522 (+ 288us) server_negotiation.cc:299] Negotiation successful
0621 05:14:52.009615 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260621 05:14:52.009989 25957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:14:51.999223 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.131.190:39053 (local address 127.0.0.1:38652)
0621 05:14:51.999593 (+ 370us) negotiation.cc:107] Waiting for socket to connect
0621 05:14:51.999651 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0621 05:14:52.000193 (+ 542us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:14:52.000480 (+ 287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:14:52.000489 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:14:52.000758 (+ 269us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:14:52.000965 (+ 207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:14:52.000969 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:14:52.001741 (+ 772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:14:52.001742 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:14:52.002603 (+ 861us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:14:52.002605 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:14:52.003023 (+ 418us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:14:52.003025 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:14:52.003219 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:14:52.003971 (+ 752us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 05:14:52.003987 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:14:52.005944 (+ 1957us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 05:14:52.008227 (+ 2283us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:14:52.008236 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:14:52.008239 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:14:52.008484 (+ 245us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:14:52.008789 (+ 305us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:14:52.008791 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:14:52.008791 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:14:52.008915 (+ 124us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:14:52.009224 (+ 309us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 05:14:52.009232 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 05:14:52.009382 (+ 150us) client_negotiation.cc:769] Sending connection context
0621 05:14:52.009499 (+ 117us) client_negotiation.cc:240] Negotiation successful
0621 05:14:52.009737 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":293,"thread_start_us":196,"threads_started":1}
I20260621 05:14:52.010795 25955 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.131.189" port: 42239 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.3.131.188@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.131.188@KRBTEST.COM" created.
Jun 21 05:14:52 dist-test-slave-vrq1 krb5kdc[25799](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782018891, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.131.189@KRBTEST.COM for kudu/127.3.131.189@KRBTEST.COM
I20260621 05:14:52.019913 25959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:14:52.011275 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49582 (local address 127.3.131.189:42239)
0621 05:14:52.011447 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0621 05:14:52.011451 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 05:14:52.011468 (+ 17us) server_negotiation.cc:407] Connection header received
0621 05:14:52.011485 (+ 17us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:14:52.011487 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:14:52.011497 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:14:52.011591 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:14:52.011867 (+ 276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:14:52.012600 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:14:52.013309 (+ 709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:14:52.013676 (+ 367us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:14:52.013769 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:14:52.016579 (+ 2810us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:14:52.016588 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:14:52.016589 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:14:52.016597 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:14:52.018435 (+ 1838us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:14:52.018885 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:14:52.018888 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:14:52.018888 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:14:52.018954 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:14:52.019243 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:14:52.019243 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:14:52.019244 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:14:52.019405 (+ 161us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:14:52.019476 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0621 05:14:52.019655 (+ 179us) server_negotiation.cc:299] Negotiation successful
0621 05:14:52.019735 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":144,"thread_start_us":86,"threads_started":1}
I20260621 05:14:52.020041 25957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:14:52.011168 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.131.189:42239 (local address 127.0.0.1:49582)
0621 05:14:52.011243 (+ 75us) negotiation.cc:107] Waiting for socket to connect
0621 05:14:52.011254 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0621 05:14:52.011298 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:14:52.011590 (+ 292us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:14:52.011591 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:14:52.011741 (+ 150us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:14:52.011791 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:14:52.011793 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:14:52.012704 (+ 911us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:14:52.012706 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:14:52.013216 (+ 510us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:14:52.013218 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:14:52.013767 (+ 549us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:14:52.013767 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:14:52.013947 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:14:52.014377 (+ 430us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 05:14:52.014381 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:14:52.016467 (+ 2086us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 05:14:52.018528 (+ 2061us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:14:52.018529 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:14:52.018529 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:14:52.018762 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:14:52.019033 (+ 271us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:14:52.019034 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:14:52.019034 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:14:52.019108 (+ 74us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:14:52.019482 (+ 374us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 05:14:52.019483 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 05:14:52.019540 (+ 57us) client_negotiation.cc:769] Sending connection context
0621 05:14:52.019629 (+ 89us) client_negotiation.cc:240] Negotiation successful
0621 05:14:52.019830 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":56,"mutex_wait_us":80}
I20260621 05:14:52.020741 25955 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.131.188" port: 35325 } has no permanent_uuid. Determining permanent_uuid...
I20260621 05:14:52.021178 25957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:14:52.021015 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.131.188:35325 (local address 127.0.0.1:53422)
0621 05:14:52.021074 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0621 05:14:52.021139 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.131.188:35325: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260621 05:14:52.021329 25904 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.131.188:35325: connect: Connection refused (error 111)
I20260621 05:14:52.021937 25957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:14:52.021817 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.131.188:35325 (local address 127.0.0.1:53424)
0621 05:14:52.021880 (+ 63us) negotiation.cc:107] Waiting for socket to connect
0621 05:14:52.021913 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.131.188:35325: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260621 05:14:52.022200 25955 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.131.188:35325: Network error: Client connection negotiation failed: client connection to 127.3.131.188:35325: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.131.188 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782018616648578-3598-0/krb5kdc/kudu_127.3.131.188.keytab.
Entry for principal kudu/127.3.131.188 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782018616648578-3598-0/krb5kdc/kudu_127.3.131.188.keytab.
I20260621 05:14:52.034569 3598 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWZOx2_/build/release/bin/kudu
/tmp/dist-test-taskWZOx2_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782018616648578-3598-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782018616648578-3598-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782018616648578-3598-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782018616648578-3598-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.131.188:35325
--webserver_interface=127.3.131.188
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782018616648578-3598-0/krb5kdc/kudu_127.3.131.188.keytab
--principal=kudu/127.3.131.188
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.131.148:44517
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.131.190:39053,127.3.131.189:42239,127.3.131.188:35325
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782018616648578-3598-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782018616648578-3598-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782018616648578-3598-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260621 05:14:52.055243 25880 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.131.189" port: 42239 } attempt: 3
Jun 21 05:14:52 dist-test-slave-vrq1 krb5kdc[25799](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782018891, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.131.190@KRBTEST.COM for kudu/127.3.131.189@KRBTEST.COM
I20260621 05:14:52.063859 25883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:14:52.055692 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.131.189:42239 (local address 127.0.0.1:49594)
0621 05:14:52.055802 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0621 05:14:52.055815 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0621 05:14:52.055866 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:14:52.056043 (+ 177us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:14:52.056044 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:14:52.056200 (+ 156us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:14:52.056266 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:14:52.056268 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:14:52.057196 (+ 928us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:14:52.057197 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:14:52.057811 (+ 614us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:14:52.057814 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:14:52.058366 (+ 552us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:14:52.058367 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:14:52.058448 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:14:52.059044 (+ 596us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 05:14:52.059050 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:14:52.060636 (+ 1586us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 05:14:52.062589 (+ 1953us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:14:52.062591 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:14:52.062591 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:14:52.062795 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:14:52.063064 (+ 269us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:14:52.063065 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:14:52.063066 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:14:52.063134 (+ 68us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:14:52.063566 (+ 432us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 05:14:52.063567 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 05:14:52.063619 (+ 52us) client_negotiation.cc:769] Sending connection context
0621 05:14:52.063686 (+ 67us) client_negotiation.cc:240] Negotiation successful
0621 05:14:52.063783 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":86}
I20260621 05:14:52.063997 25959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:14:52.055779 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49594 (local address 127.3.131.189:42239)
0621 05:14:52.055842 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0621 05:14:52.055845 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0621 05:14:52.055857 (+ 12us) server_negotiation.cc:407] Connection header received
0621 05:14:52.055925 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:14:52.055926 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:14:52.055937 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:14:52.056022 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:14:52.056341 (+ 319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:14:52.057087 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:14:52.057903 (+ 816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:14:52.058257 (+ 354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:14:52.058332 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:14:52.060743 (+ 2411us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:14:52.060744 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:14:52.060745 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:14:52.060754 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:14:52.062470 (+ 1716us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:14:52.062881 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:14:52.062882 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:14:52.062882 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:14:52.062952 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:14:52.063286 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:14:52.063287 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:14:52.063288 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:14:52.063447 (+ 159us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:14:52.063519 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0621 05:14:52.063730 (+ 211us) server_negotiation.cc:299] Negotiation successful
0621 05:14:52.063905 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260621 05:14:52.064500 25880 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.131.188" port: 35325 } has no permanent_uuid. Determining permanent_uuid...
I20260621 05:14:52.064949 25883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:14:52.064832 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.131.188:35325 (local address 127.0.0.1:53430)
0621 05:14:52.064875 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0621 05:14:52.064916 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.131.188:35325: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":20}
I20260621 05:14:52.065761 25883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:14:52.065622 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.131.188:35325 (local address 127.0.0.1:53434)
0621 05:14:52.065690 (+ 68us) negotiation.cc:107] Waiting for socket to connect
0621 05:14:52.065728 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.131.188:35325: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260621 05:14:52.065881 25880 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.131.188:35325: Network error: Client connection negotiation failed: client connection to 127.3.131.188:35325: connect: Connection refused (error 111)
I20260621 05:14:52.071465 25955 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.131.188" port: 35325 } attempt: 1
I20260621 05:14:52.072021 25957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:14:52.071850 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.131.188:35325 (local address 127.0.0.1:53446)
0621 05:14:52.071920 (+ 70us) negotiation.cc:107] Waiting for socket to connect
0621 05:14:52.071988 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.131.188:35325: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20260621 05:14:52.072382 25957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:14:52.072286 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.131.188:35325 (local address 127.0.0.1:53448)
0621 05:14:52.072332 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0621 05:14:52.072358 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.131.188:35325: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260621 05:14:52.072472 25955 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.131.188:35325: Network error: Client connection negotiation failed: client connection to 127.3.131.188:35325: connect: Connection refused (error 111)
I20260621 05:14:52.097054 25880 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.131.188" port: 35325 } attempt: 1
I20260621 05:14:52.097611 25883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:14:52.097452 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.131.188:35325 (local address 127.0.0.1:53456)
0621 05:14:52.097517 (+ 65us) negotiation.cc:107] Waiting for socket to connect
0621 05:14:52.097567 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.131.188:35325: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
I20260621 05:14:52.098022 25883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:14:52.097912 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.131.188:35325 (local address 127.0.0.1:53460)
0621 05:14:52.097970 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0621 05:14:52.097999 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.131.188:35325: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260621 05:14:52.098165 25880 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.131.188:35325: Network error: Client connection negotiation failed: client connection to 127.3.131.188:35325: connect: Connection refused (error 111)
W20260621 05:14:52.122783 25962 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260621 05:14:52.123028 25962 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 05:14:52.123046 25962 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 05:14:52.123068 25962 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 05:14:52.124398 25962 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 05:14:52.124457 25962 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 05:14:52.124472 25962 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 05:14:52.124486 25962 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 05:14:52.124498 25962 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 05:14:52.126049 25962 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.131.148:44517
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782018616648578-3598-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782018616648578-3598-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.3.131.190:39053,127.3.131.189:42239,127.3.131.188:35325
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782018616648578-3598-0/krb5kdc/kudu_127.3.131.188.keytab
--principal=kudu/127.3.131.188
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.131.188:35325
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782018616648578-3598-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.3.131.188
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25962
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782018616648578-3598-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 21 Jun 2026 04:16:16 UTC on ccycloud.quasar-xycaha.root.comops.site
build id 4457881
I20260621 05:14:52.126276 25962 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 05:14:52.126461 25962 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 05:14:52.131839 25968 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:14:52.131826 25970 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:14:52.131888 25967 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:14:52.132275 25962 server_base.cc:1034] running on GCE node
I20260621 05:14:52.132592 25962 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 05:14:52.132930 25962 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 05:14:52.134140 25962 hybrid_clock.cc:648] HybridClock initialized: now 1782018892134079 us; error 71 us; skew 500 ppm
Jun 21 05:14:52 dist-test-slave-vrq1 krb5kdc[25799](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782018892, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.131.188@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 05:14:52.137436 25962 init.cc:377] Logged in from keytab as kudu/127.3.131.188@KRBTEST.COM (short username kudu)
I20260621 05:14:52.137975 25962 webserver.cc:460] Webserver started at http://127.3.131.188:35079/ using document root <none> and password file <none>
I20260621 05:14:52.138242 25962 fs_manager.cc:359] Metadata directory not provided
I20260621 05:14:52.138293 25962 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 05:14:52.138434 25962 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 05:14:52.139678 25962 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782018616648578-3598-0/minicluster-data/master-2/data/instance:
uuid: "8fb5e91e5845433999a13248ad4721ab"
format_stamp: "Formatted at 2026-06-21 05:14:52 on dist-test-slave-vrq1"
server_key: "e081ac54a238d50d1a4722e6c1095a2d"
server_key_iv: "48414d3e45ab5b88d2fedb7a8636a630"
server_key_version: "encryptionkey@0"
I20260621 05:14:52.140057 25962 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782018616648578-3598-0/minicluster-data/master-2/wal/instance:
uuid: "8fb5e91e5845433999a13248ad4721ab"
format_stamp: "Formatted at 2026-06-21 05:14:52 on dist-test-slave-vrq1"
server_key: "e081ac54a238d50d1a4722e6c1095a2d"
server_key_iv: "48414d3e45ab5b88d2fedb7a8636a630"
server_key_version: "encryptionkey@0"
I20260621 05:14:52.140637 25955 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.131.188" port: 35325 } attempt: 2
I20260621 05:14:52.141165 25957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:14:52.141003 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.131.188:35325 (local address 127.0.0.1:53462)
0621 05:14:52.141079 (+ 76us) negotiation.cc:107] Waiting for socket to connect
0621 05:14:52.141121 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.131.188:35325: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260621 05:14:52.141655 25957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:14:52.141550 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.131.188:35325 (local address 127.0.0.1:53474)
0621 05:14:52.141599 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0621 05:14:52.141633 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.131.188:35325: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
W20260621 05:14:52.141805 25955 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.131.188:35325: Network error: Client connection negotiation failed: client connection to 127.3.131.188:35325: connect: Connection refused (error 111)
I20260621 05:14:52.142025 25962 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.002s sys 0.000s
I20260621 05:14:52.142822 25977 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 05:14:52.143046 25962 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260621 05:14:52.143240 25962 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782018616648578-3598-0/minicluster-data/master-2/data,/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782018616648578-3598-0/minicluster-data/master-2/wal
uuid: "8fb5e91e5845433999a13248ad4721ab"
format_stamp: "Formatted at 2026-06-21 05:14:52 on dist-test-slave-vrq1"
server_key: "e081ac54a238d50d1a4722e6c1095a2d"
server_key_iv: "48414d3e45ab5b88d2fedb7a8636a630"
server_key_version: "encryptionkey@0"
I20260621 05:14:52.143334 25962 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782018616648578-3598-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782018616648578-3598-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782018616648578-3598-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 05:14:52.173332 25880 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.131.188" port: 35325 } attempt: 2
I20260621 05:14:52.173790 25883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:14:52.173649 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.131.188:35325 (local address 127.0.0.1:53482)
0621 05:14:52.173715 (+ 66us) negotiation.cc:107] Waiting for socket to connect
0621 05:14:52.173754 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.131.188:35325: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
I20260621 05:14:52.174244 25883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:14:52.174087 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.131.188:35325 (local address 127.0.0.1:53492)
0621 05:14:52.174177 (+ 90us) negotiation.cc:107] Waiting for socket to connect
0621 05:14:52.174219 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.131.188:35325: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260621 05:14:52.174379 25880 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.131.188:35325: Network error: Client connection negotiation failed: client connection to 127.3.131.188:35325: connect: Connection refused (error 111)
I20260621 05:14:52.175148 25962 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 05:14:52.177392 25962 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 05:14:52.177559 25962 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 05:14:52.181993 25962 rpc_server.cc:307] RPC server started. Bound to: 127.3.131.188:35325
I20260621 05:14:52.182026 26029 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.131.188:35325 every 8 connection(s)
I20260621 05:14:52.182391 25962 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782018616648578-3598-0/minicluster-data/master-2/data/info.pb
I20260621 05:14:52.183471 26030 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 05:14:52.185007 26030 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.131.190" port: 39053 } has no permanent_uuid. Determining permanent_uuid...
I20260621 05:14:52.190441 3598 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWZOx2_/build/release/bin/kudu as pid 25962
I20260621 05:14:52.190529 3598 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782018616648578-3598-0/minicluster-data/master-2/wal/instance
I20260621 05:14:52.190717 3598 external_mini_cluster.cc:1428] Setting key caab867e8812ff27306d08cceb237007
Jun 21 05:14:52 dist-test-slave-vrq1 krb5kdc[25799](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782018892, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.131.188@KRBTEST.COM for kudu/127.3.131.190@KRBTEST.COM
I20260621 05:14:52.195459 25882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:14:52.185750 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38660 (local address 127.3.131.190:39053)
0621 05:14:52.185825 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0621 05:14:52.185827 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:14:52.186056 (+ 229us) server_negotiation.cc:407] Connection header received
0621 05:14:52.186512 (+ 456us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:14:52.186513 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:14:52.186525 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:14:52.186599 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:14:52.187268 (+ 669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:14:52.187834 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:14:52.188814 (+ 980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:14:52.189041 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:14:52.189117 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:14:52.191973 (+ 2856us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:14:52.191975 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:14:52.191975 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:14:52.191982 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:14:52.193763 (+ 1781us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:14:52.194279 (+ 516us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:14:52.194280 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:14:52.194281 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:14:52.194333 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:14:52.194690 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:14:52.194691 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:14:52.194691 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:14:52.194853 (+ 162us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:14:52.194923 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0621 05:14:52.195297 (+ 374us) server_negotiation.cc:299] Negotiation successful
0621 05:14:52.195394 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260621 05:14:52.196074 26032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:14:52.185731 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.131.190:39053 (local address 127.0.0.1:38660)
0621 05:14:52.185898 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0621 05:14:52.185945 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0621 05:14:52.186387 (+ 442us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:14:52.186665 (+ 278us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:14:52.186668 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:14:52.186914 (+ 246us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:14:52.187109 (+ 195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:14:52.187113 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:14:52.187924 (+ 811us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:14:52.187925 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:14:52.188719 (+ 794us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:14:52.188721 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:14:52.189144 (+ 423us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:14:52.189145 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:14:52.189247 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:14:52.189790 (+ 543us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 05:14:52.189798 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:14:52.191851 (+ 2053us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 05:14:52.193860 (+ 2009us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:14:52.193867 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:14:52.193869 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:14:52.194191 (+ 322us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:14:52.194395 (+ 204us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:14:52.194396 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:14:52.194397 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:14:52.194546 (+ 149us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:14:52.194927 (+ 381us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 05:14:52.194932 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 05:14:52.195062 (+ 130us) client_negotiation.cc:769] Sending connection context
0621 05:14:52.195272 (+ 210us) client_negotiation.cc:240] Negotiation successful
0621 05:14:52.195453 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":92,"threads_started":1}
I20260621 05:14:52.196738 26030 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.131.189" port: 42239 } has no permanent_uuid. Determining permanent_uuid...
Jun 21 05:14:52 dist-test-slave-vrq1 krb5kdc[25799](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782018891, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.131.188@KRBTEST.COM
Jun 21 05:14:52 dist-test-slave-vrq1 krb5kdc[25799](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782018892, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.131.188@KRBTEST.COM for kudu/127.3.131.189@KRBTEST.COM
I20260621 05:14:52.201431 26034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:14:52.191212 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53496 (local address 127.3.131.188:35325)
0621 05:14:52.191371 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0621 05:14:52.191375 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 05:14:52.191441 (+ 66us) server_negotiation.cc:407] Connection header received
0621 05:14:52.191519 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:14:52.191521 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:14:52.191538 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:14:52.191602 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:14:52.192478 (+ 876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:14:52.193589 (+ 1111us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:14:52.194744 (+ 1155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:14:52.195142 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:14:52.195285 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:14:52.198049 (+ 2764us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:14:52.198056 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:14:52.198057 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:14:52.198063 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:14:52.199949 (+ 1886us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:14:52.200338 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:14:52.200341 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:14:52.200341 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:14:52.200397 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:14:52.200702 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:14:52.200703 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:14:52.200703 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:14:52.200842 (+ 139us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:14:52.200957 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0621 05:14:52.201125 (+ 168us) server_negotiation.cc:299] Negotiation successful
0621 05:14:52.201315 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":120,"thread_start_us":88,"threads_started":1}
I20260621 05:14:52.204944 26032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:14:52.197061 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.131.189:42239 (local address 127.0.0.1:49600)
0621 05:14:52.197187 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0621 05:14:52.197198 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0621 05:14:52.197253 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:14:52.197376 (+ 123us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:14:52.197377 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:14:52.197515 (+ 138us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:14:52.197572 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:14:52.197574 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:14:52.198349 (+ 775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:14:52.198350 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:14:52.199040 (+ 690us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:14:52.199043 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:14:52.199573 (+ 530us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:14:52.199574 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:14:52.199642 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:14:52.200080 (+ 438us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 05:14:52.200084 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:14:52.201513 (+ 1429us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 05:14:52.204003 (+ 2490us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:14:52.204003 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:14:52.204004 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:14:52.204175 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:14:52.204405 (+ 230us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:14:52.204405 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:14:52.204405 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:14:52.204447 (+ 42us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:14:52.204686 (+ 239us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 05:14:52.204686 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 05:14:52.204718 (+ 32us) client_negotiation.cc:769] Sending connection context
0621 05:14:52.204764 (+ 46us) client_negotiation.cc:240] Negotiation successful
0621 05:14:52.204831 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":107}
I20260621 05:14:52.204973 25959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:14:52.197136 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49600 (local address 127.3.131.189:42239)
0621 05:14:52.197190 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0621 05:14:52.197192 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:14:52.197237 (+ 45us) server_negotiation.cc:407] Connection header received
0621 05:14:52.197317 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:14:52.197317 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:14:52.197326 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:14:52.197371 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:14:52.197658 (+ 287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:14:52.198222 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:14:52.199158 (+ 936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:14:52.199486 (+ 328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:14:52.199569 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:14:52.201635 (+ 2066us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:14:52.201636 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:14:52.201636 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:14:52.201645 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:14:52.203333 (+ 1688us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:14:52.204259 (+ 926us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:14:52.204260 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:14:52.204260 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:14:52.204333 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:14:52.204501 (+ 168us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:14:52.204502 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:14:52.204502 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:14:52.204627 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:14:52.204690 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0621 05:14:52.204787 (+ 97us) server_negotiation.cc:299] Negotiation successful
0621 05:14:52.204868 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260621 05:14:52.205751 26030 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.131.188" port: 35325 } has no permanent_uuid. Determining permanent_uuid...
Jun 21 05:14:52 dist-test-slave-vrq1 krb5kdc[25799](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782018891, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.131.190@KRBTEST.COM
Jun 21 05:14:52 dist-test-slave-vrq1 krb5kdc[25799](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782018892, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.131.188@KRBTEST.COM for kudu/127.3.131.188@KRBTEST.COM
I20260621 05:14:52.210667 25882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:14:52.202487 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38662 (local address 127.3.131.190:39053)
0621 05:14:52.202667 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0621 05:14:52.202668 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0621 05:14:52.202675 (+ 7us) server_negotiation.cc:407] Connection header received
0621 05:14:52.202695 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:14:52.202696 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:14:52.202704 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:14:52.202788 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:14:52.203424 (+ 636us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:14:52.203916 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:14:52.204921 (+ 1005us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:14:52.205206 (+ 285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:14:52.205260 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:14:52.207641 (+ 2381us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:14:52.207642 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:14:52.207643 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:14:52.207650 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:14:52.209281 (+ 1631us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:14:52.209694 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:14:52.209695 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:14:52.209695 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:14:52.209747 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:14:52.209992 (+ 245us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:14:52.209993 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:14:52.209993 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:14:52.210159 (+ 166us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:14:52.210296 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0621 05:14:52.210408 (+ 112us) server_negotiation.cc:299] Negotiation successful
0621 05:14:52.210553 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":122}
I20260621 05:14:52.213580 26032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:14:52.206031 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.131.188:35325 (local address 127.0.0.1:53512)
0621 05:14:52.206080 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0621 05:14:52.206088 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0621 05:14:52.206135 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:14:52.206238 (+ 103us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:14:52.206238 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:14:52.206364 (+ 126us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:14:52.206390 (+ 26us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:14:52.206392 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:14:52.207331 (+ 939us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:14:52.207331 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:14:52.207951 (+ 620us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:14:52.207954 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:14:52.208367 (+ 413us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:14:52.208368 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:14:52.208428 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:14:52.208681 (+ 253us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 05:14:52.208686 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:14:52.210489 (+ 1803us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 05:14:52.212314 (+ 1825us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:14:52.212315 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:14:52.212315 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:14:52.212495 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:14:52.212861 (+ 366us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:14:52.212862 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:14:52.212863 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:14:52.212938 (+ 75us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:14:52.213235 (+ 297us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 05:14:52.213235 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 05:14:52.213283 (+ 48us) client_negotiation.cc:769] Sending connection context
0621 05:14:52.213364 (+ 81us) client_negotiation.cc:240] Negotiation successful
0621 05:14:52.213462 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":29}
I20260621 05:14:52.213873 26034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:14:52.206085 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53512 (local address 127.3.131.188:35325)
0621 05:14:52.206143 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0621 05:14:52.206144 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0621 05:14:52.206155 (+ 11us) server_negotiation.cc:407] Connection header received
0621 05:14:52.206184 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:14:52.206185 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:14:52.206194 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:14:52.206238 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:14:52.206468 (+ 230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:14:52.207241 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:14:52.208055 (+ 814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:14:52.208294 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:14:52.208346 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:14:52.210606 (+ 2260us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:14:52.210607 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:14:52.210608 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:14:52.210614 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:14:52.212205 (+ 1591us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:14:52.212659 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:14:52.212660 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:14:52.212661 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:14:52.212721 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:14:52.213038 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:14:52.213040 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:14:52.213040 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:14:52.213155 (+ 115us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:14:52.213221 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0621 05:14:52.213427 (+ 206us) server_negotiation.cc:299] Negotiation successful
0621 05:14:52.213626 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":108,"server-negotiator.queue_time_us":41}
I20260621 05:14:52.215893 26030 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8fb5e91e5845433999a13248ad4721ab: Bootstrap starting.
I20260621 05:14:52.217010 26030 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8fb5e91e5845433999a13248ad4721ab: Neither blocks nor log segments found. Creating new log.
Jun 21 05:14:52 dist-test-slave-vrq1 krb5kdc[25799](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782018891, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.131.189@KRBTEST.COM
I20260621 05:14:52.217357 26030 log.cc:824] T 00000000000000000000000000000000 P 8fb5e91e5845433999a13248ad4721ab: Log is configured to *not* fsync() on all Append() calls
I20260621 05:14:52.218327 26030 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8fb5e91e5845433999a13248ad4721ab: No bootstrap required, opened a new log
I20260621 05:14:52.220407 26030 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8fb5e91e5845433999a13248ad4721ab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7023899fb034854a56325cab379fabe" member_type: VOTER last_known_addr { host: "127.3.131.190" port: 39053 } } peers { permanent_uuid: "c4e36f6e2d6b4e2f9e95bdf4b99f2d08" member_type: VOTER last_known_addr { host: "127.3.131.189" port: 42239 } } peers { permanent_uuid: "8fb5e91e5845433999a13248ad4721ab" member_type: VOTER last_known_addr { host: "127.3.131.188" port: 35325 } }
I20260621 05:14:52.220589 26030 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8fb5e91e5845433999a13248ad4721ab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8fb5e91e5845433999a13248ad4721ab, State: Initialized, Role: FOLLOWER
I20260621 05:14:52.220712 26030 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8fb5e91e5845433999a13248ad4721ab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7023899fb034854a56325cab379fabe" member_type: VOTER last_known_addr { host: "127.3.131.190" port: 39053 } } peers { permanent_uuid: "c4e36f6e2d6b4e2f9e95bdf4b99f2d08" member_type: VOTER last_known_addr { host: "127.3.131.189" port: 42239 } } peers { permanent_uuid: "8fb5e91e5845433999a13248ad4721ab" member_type: VOTER last_known_addr { host: "127.3.131.188" port: 35325 } }
I20260621 05:14:52.220881 25959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:14:52.211494 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49602 (local address 127.3.131.189:42239)
0621 05:14:52.211536 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0621 05:14:52.211538 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:14:52.211551 (+ 13us) server_negotiation.cc:407] Connection header received
0621 05:14:52.211577 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:14:52.211578 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:14:52.211587 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:14:52.211655 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:14:52.212147 (+ 492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:14:52.212774 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:14:52.213794 (+ 1020us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:14:52.214221 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:14:52.214265 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:14:52.217850 (+ 3585us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:14:52.217852 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:14:52.217852 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:14:52.217861 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:14:52.219563 (+ 1702us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:14:52.219941 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:14:52.219942 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:14:52.219943 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:14:52.220010 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:14:52.220224 (+ 214us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:14:52.220225 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:14:52.220225 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:14:52.220349 (+ 124us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:14:52.220531 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0621 05:14:52.220571 (+ 40us) server_negotiation.cc:299] Negotiation successful
0621 05:14:52.220659 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260621 05:14:52.221062 26036 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8fb5e91e5845433999a13248ad4721ab [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7023899fb034854a56325cab379fabe" member_type: VOTER last_known_addr { host: "127.3.131.190" port: 39053 } } peers { permanent_uuid: "c4e36f6e2d6b4e2f9e95bdf4b99f2d08" member_type: VOTER last_known_addr { host: "127.3.131.189" port: 42239 } } peers { permanent_uuid: "8fb5e91e5845433999a13248ad4721ab" member_type: VOTER last_known_addr { host: "127.3.131.188" port: 35325 } } }
I20260621 05:14:52.221150 26030 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8fb5e91e5845433999a13248ad4721ab [sys.catalog]: configured and running, proceeding with master startup.
I20260621 05:14:52.221172 26036 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8fb5e91e5845433999a13248ad4721ab [sys.catalog]: This master's current role is: FOLLOWER
I20260621 05:14:52.222698 3598 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
W20260621 05:14:52.223239 26047 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 8fb5e91e5845433999a13248ad4721ab: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260621 05:14:52.223322 26047 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260621 05:14:52.224025 25854 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "f7023899fb034854a56325cab379fabe"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38662
I20260621 05:14:52.223753 25962 master_runner.cc:186] Error getting master registration for 127.3.131.190:39053: OK, instance_id { permanent_uuid: "f7023899fb034854a56325cab379fabe" instance_seqno: 1782018891826028 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260621 05:14:52.225962 25962 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260621 05:14:52.232977 25955 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.131.188" port: 35325 } attempt: 3
Jun 21 05:14:52 dist-test-slave-vrq1 krb5kdc[25799](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782018891, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.131.189@KRBTEST.COM for kudu/127.3.131.188@KRBTEST.COM
I20260621 05:14:52.242053 26034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:14:52.233466 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53524 (local address 127.3.131.188:35325)
0621 05:14:52.233534 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0621 05:14:52.233536 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:14:52.233548 (+ 12us) server_negotiation.cc:407] Connection header received
0621 05:14:52.233679 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:14:52.233680 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:14:52.233692 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:14:52.233756 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:14:52.234165 (+ 409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:14:52.234964 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:14:52.236140 (+ 1176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:14:52.236482 (+ 342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:14:52.236564 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:14:52.238998 (+ 2434us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:14:52.239000 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:14:52.239000 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:14:52.239008 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:14:52.240750 (+ 1742us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:14:52.241174 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:14:52.241175 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:14:52.241175 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:14:52.241238 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:14:52.241452 (+ 214us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:14:52.241453 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:14:52.241453 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:14:52.241652 (+ 199us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:14:52.241721 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0621 05:14:52.241875 (+ 154us) server_negotiation.cc:299] Negotiation successful
0621 05:14:52.241956 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260621 05:14:52.242056 25957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:14:52.233373 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.131.188:35325 (local address 127.0.0.1:53524)
0621 05:14:52.233458 (+ 85us) negotiation.cc:107] Waiting for socket to connect
0621 05:14:52.233473 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0621 05:14:52.233544 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:14:52.233778 (+ 234us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:14:52.233779 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:14:52.233990 (+ 211us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:14:52.234051 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:14:52.234054 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:14:52.235060 (+ 1006us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:14:52.235061 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:14:52.236037 (+ 976us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:14:52.236039 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:14:52.236586 (+ 547us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:14:52.236587 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:14:52.236656 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:14:52.237242 (+ 586us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 05:14:52.237248 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:14:52.238900 (+ 1652us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 05:14:52.240855 (+ 1955us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:14:52.240857 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:14:52.240857 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:14:52.241099 (+ 242us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:14:52.241309 (+ 210us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:14:52.241310 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:14:52.241311 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:14:52.241374 (+ 63us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:14:52.241718 (+ 344us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 05:14:52.241719 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 05:14:52.241768 (+ 49us) client_negotiation.cc:769] Sending connection context
0621 05:14:52.241836 (+ 68us) client_negotiation.cc:240] Negotiation successful
0621 05:14:52.241937 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":38}
I20260621 05:14:52.244133 25955 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c4e36f6e2d6b4e2f9e95bdf4b99f2d08: Bootstrap starting.
I20260621 05:14:52.244882 25955 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c4e36f6e2d6b4e2f9e95bdf4b99f2d08: Neither blocks nor log segments found. Creating new log.
I20260621 05:14:52.245184 25955 log.cc:824] T 00000000000000000000000000000000 P c4e36f6e2d6b4e2f9e95bdf4b99f2d08: Log is configured to *not* fsync() on all Append() calls
I20260621 05:14:52.246037 25955 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c4e36f6e2d6b4e2f9e95bdf4b99f2d08: No bootstrap required, opened a new log
I20260621 05:14:52.247778 25955 raft_consensus.cc:348] T 00000000000000000000000000000000 P c4e36f6e2d6b4e2f9e95bdf4b99f2d08 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7023899fb034854a56325cab379fabe" member_type: VOTER last_known_addr { host: "127.3.131.190" port: 39053 } } peers { permanent_uuid: "c4e36f6e2d6b4e2f9e95bdf4b99f2d08" member_type: VOTER last_known_addr { host: "127.3.131.189" port: 42239 } } peers { permanent_uuid: "8fb5e91e5845433999a13248ad4721ab" member_type: VOTER last_known_addr { host: "127.3.131.188" port: 35325 } }
I20260621 05:14:52.247920 25955 raft_consensus.cc:729] T 00000000000000000000000000000000 P c4e36f6e2d6b4e2f9e95bdf4b99f2d08 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c4e36f6e2d6b4e2f9e95bdf4b99f2d08, State: Initialized, Role: FOLLOWER
I20260621 05:14:52.248009 25955 consensus_queue.cc:260] T 00000000000000000000000000000000 P c4e36f6e2d6b4e2f9e95bdf4b99f2d08 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7023899fb034854a56325cab379fabe" member_type: VOTER last_known_addr { host: "127.3.131.190" port: 39053 } } peers { permanent_uuid: "c4e36f6e2d6b4e2f9e95bdf4b99f2d08" member_type: VOTER last_known_addr { host: "127.3.131.189" port: 42239 } } peers { permanent_uuid: "8fb5e91e5845433999a13248ad4721ab" member_type: VOTER last_known_addr { host: "127.3.131.188" port: 35325 } }
I20260621 05:14:52.248301 26050 sys_catalog.cc:455] T 00000000000000000000000000000000 P c4e36f6e2d6b4e2f9e95bdf4b99f2d08 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7023899fb034854a56325cab379fabe" member_type: VOTER last_known_addr { host: "127.3.131.190" port: 39053 } } peers { permanent_uuid: "c4e36f6e2d6b4e2f9e95bdf4b99f2d08" member_type: VOTER last_known_addr { host: "127.3.131.189" port: 42239 } } peers { permanent_uuid: "8fb5e91e5845433999a13248ad4721ab" member_type: VOTER last_known_addr { host: "127.3.131.188" port: 35325 } } }
I20260621 05:14:52.248380 25955 sys_catalog.cc:564] T 00000000000000000000000000000000 P c4e36f6e2d6b4e2f9e95bdf4b99f2d08 [sys.catalog]: configured and running, proceeding with master startup.
I20260621 05:14:52.248380 26050 sys_catalog.cc:458] T 00000000000000000000000000000000 P c4e36f6e2d6b4e2f9e95bdf4b99f2d08 [sys.catalog]: This master's current role is: FOLLOWER
W20260621 05:14:52.250154 26061 catalog_manager.cc:1558] T 00000000000000000000000000000000 P c4e36f6e2d6b4e2f9e95bdf4b99f2d08: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260621 05:14:52.250211 26061 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260621 05:14:52.250473 25887 master_runner.cc:186] Error getting master registration for 127.3.131.190:39053: OK, instance_id { permanent_uuid: "f7023899fb034854a56325cab379fabe" instance_seqno: 1782018891826028 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260621 05:14:52.253335 25887 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260621 05:14:52.273533 25880 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.131.188" port: 35325 } attempt: 3
Jun 21 05:14:52 dist-test-slave-vrq1 krb5kdc[25799](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782018891, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.131.190@KRBTEST.COM for kudu/127.3.131.188@KRBTEST.COM
I20260621 05:14:52.281426 25883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:14:52.273936 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.131.188:35325 (local address 127.0.0.1:53534)
0621 05:14:52.274001 (+ 65us) negotiation.cc:107] Waiting for socket to connect
0621 05:14:52.274012 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0621 05:14:52.274067 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:14:52.274346 (+ 279us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:14:52.274347 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:14:52.274574 (+ 227us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:14:52.274631 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:14:52.274633 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:14:52.275399 (+ 766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:14:52.275401 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:14:52.276212 (+ 811us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:14:52.276214 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:14:52.276596 (+ 382us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:14:52.276597 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:14:52.276656 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:14:52.277178 (+ 522us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 05:14:52.277183 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:14:52.278615 (+ 1432us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 05:14:52.280308 (+ 1693us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:14:52.280309 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:14:52.280310 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:14:52.280536 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:14:52.280731 (+ 195us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:14:52.280732 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:14:52.280732 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:14:52.280796 (+ 64us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:14:52.281080 (+ 284us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 05:14:52.281081 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 05:14:52.281127 (+ 46us) client_negotiation.cc:769] Sending connection context
0621 05:14:52.281220 (+ 93us) client_negotiation.cc:240] Negotiation successful
0621 05:14:52.281326 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":43}
I20260621 05:14:52.281426 26034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:14:52.274160 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53534 (local address 127.3.131.188:35325)
0621 05:14:52.274208 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0621 05:14:52.274210 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:14:52.274218 (+ 8us) server_negotiation.cc:407] Connection header received
0621 05:14:52.274250 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:14:52.274251 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:14:52.274259 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:14:52.274324 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:14:52.274698 (+ 374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:14:52.275297 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:14:52.276297 (+ 1000us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:14:52.276499 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:14:52.276593 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:14:52.278688 (+ 2095us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:14:52.278690 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:14:52.278691 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:14:52.278699 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:14:52.280196 (+ 1497us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:14:52.280598 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:14:52.280599 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:14:52.280599 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:14:52.280660 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:14:52.280849 (+ 189us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:14:52.280850 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:14:52.280850 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:14:52.281001 (+ 151us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:14:52.281078 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0621 05:14:52.281248 (+ 170us) server_negotiation.cc:299] Negotiation successful
0621 05:14:52.281327 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260621 05:14:52.283538 25880 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f7023899fb034854a56325cab379fabe: Bootstrap starting.
I20260621 05:14:52.284318 25880 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f7023899fb034854a56325cab379fabe: Neither blocks nor log segments found. Creating new log.
I20260621 05:14:52.284641 25880 log.cc:824] T 00000000000000000000000000000000 P f7023899fb034854a56325cab379fabe: Log is configured to *not* fsync() on all Append() calls
I20260621 05:14:52.285466 25880 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f7023899fb034854a56325cab379fabe: No bootstrap required, opened a new log
I20260621 05:14:52.286834 25880 raft_consensus.cc:348] T 00000000000000000000000000000000 P f7023899fb034854a56325cab379fabe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7023899fb034854a56325cab379fabe" member_type: VOTER last_known_addr { host: "127.3.131.190" port: 39053 } } peers { permanent_uuid: "c4e36f6e2d6b4e2f9e95bdf4b99f2d08" member_type: VOTER last_known_addr { host: "127.3.131.189" port: 42239 } } peers { permanent_uuid: "8fb5e91e5845433999a13248ad4721ab" member_type: VOTER last_known_addr { host: "127.3.131.188" port: 35325 } }
I20260621 05:14:52.286979 25880 raft_consensus.cc:729] T 00000000000000000000000000000000 P f7023899fb034854a56325cab379fabe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f7023899fb034854a56325cab379fabe, State: Initialized, Role: FOLLOWER
I20260621 05:14:52.287073 25880 consensus_queue.cc:260] T 00000000000000000000000000000000 P f7023899fb034854a56325cab379fabe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7023899fb034854a56325cab379fabe" member_type: VOTER last_known_addr { host: "127.3.131.190" port: 39053 } } peers { permanent_uuid: "c4e36f6e2d6b4e2f9e95bdf4b99f2d08" member_type: VOTER last_known_addr { host: "127.3.131.189" port: 42239 } } peers { permanent_uuid: "8fb5e91e5845433999a13248ad4721ab" member_type: VOTER last_known_addr { host: "127.3.131.188" port: 35325 } }
I20260621 05:14:52.287428 25880 sys_catalog.cc:564] T 00000000000000000000000000000000 P f7023899fb034854a56325cab379fabe [sys.catalog]: configured and running, proceeding with master startup.
I20260621 05:14:52.287442 25854 raft_consensus.cc:482] T 00000000000000000000000000000000 P f7023899fb034854a56325cab379fabe [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260621 05:14:52.287490 25854 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f7023899fb034854a56325cab379fabe [term 0 FOLLOWER]: Advancing to term 1
I20260621 05:14:52.288414 25854 raft_consensus.cc:504] T 00000000000000000000000000000000 P f7023899fb034854a56325cab379fabe [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7023899fb034854a56325cab379fabe" member_type: VOTER last_known_addr { host: "127.3.131.190" port: 39053 } } peers { permanent_uuid: "c4e36f6e2d6b4e2f9e95bdf4b99f2d08" member_type: VOTER last_known_addr { host: "127.3.131.189" port: 42239 } } peers { permanent_uuid: "8fb5e91e5845433999a13248ad4721ab" member_type: VOTER last_known_addr { host: "127.3.131.188" port: 35325 } }
I20260621 05:14:52.289361 25929 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f7023899fb034854a56325cab379fabe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "c4e36f6e2d6b4e2f9e95bdf4b99f2d08"
I20260621 05:14:52.289517 25929 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c4e36f6e2d6b4e2f9e95bdf4b99f2d08 [term 0 FOLLOWER]: Advancing to term 1
I20260621 05:14:52.290501 25929 raft_consensus.cc:2454] T 00000000000000000000000000000000 P c4e36f6e2d6b4e2f9e95bdf4b99f2d08 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f7023899fb034854a56325cab379fabe in term 1.
I20260621 05:14:52.290766 26004 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f7023899fb034854a56325cab379fabe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "8fb5e91e5845433999a13248ad4721ab"
I20260621 05:14:52.290884 26004 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8fb5e91e5845433999a13248ad4721ab [term 0 FOLLOWER]: Advancing to term 1
I20260621 05:14:52.291780 26004 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 8fb5e91e5845433999a13248ad4721ab [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f7023899fb034854a56325cab379fabe in term 1.
I20260621 05:14:52.292449 25831 leader_election.cc:302] T 00000000000000000000000000000000 P f7023899fb034854a56325cab379fabe [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c4e36f6e2d6b4e2f9e95bdf4b99f2d08, f7023899fb034854a56325cab379fabe; no voters:
I20260621 05:14:52.288702 25854 leader_election.cc:288] T 00000000000000000000000000000000 P f7023899fb034854a56325cab379fabe [CANDIDATE]: Term 1 election: Requested vote from peers c4e36f6e2d6b4e2f9e95bdf4b99f2d08 (127.3.131.189:42239), 8fb5e91e5845433999a13248ad4721ab (127.3.131.188:35325)
I20260621 05:14:52.295338 25812 master_runner.cc:263] No leader master found from master f7023899fb034854a56325cab379fabe
I20260621 05:14:52.295388 25812 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260621 05:14:52.289680 26075 catalog_manager.cc:1558] T 00000000000000000000000000000000 P f7023899fb034854a56325cab379fabe: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260621 05:14:52.295542 26075 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260621 05:14:52.287395 26064 sys_catalog.cc:455] T 00000000000000000000000000000000 P f7023899fb034854a56325cab379fabe [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7023899fb034854a56325cab379fabe" member_type: VOTER last_known_addr { host: "127.3.131.190" port: 39053 } } peers { permanent_uuid: "c4e36f6e2d6b4e2f9e95bdf4b99f2d08" member_type: VOTER last_known_addr { host: "127.3.131.189" port: 42239 } } peers { permanent_uuid: "8fb5e91e5845433999a13248ad4721ab" member_type: VOTER last_known_addr { host: "127.3.131.188" port: 35325 } } }
I20260621 05:14:52.295763 26064 sys_catalog.cc:458] T 00000000000000000000000000000000 P f7023899fb034854a56325cab379fabe [sys.catalog]: This master's current role is: FOLLOWER
I20260621 05:14:52.295851 26064 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f7023899fb034854a56325cab379fabe [term 1 FOLLOWER]: Leader election won for term 1
I20260621 05:14:52.295957 26064 raft_consensus.cc:686] T 00000000000000000000000000000000 P f7023899fb034854a56325cab379fabe [term 1 LEADER]: Becoming Leader. State: Replica: f7023899fb034854a56325cab379fabe, State: Running, Role: LEADER
I20260621 05:14:52.296085 26064 consensus_queue.cc:237] T 00000000000000000000000000000000 P f7023899fb034854a56325cab379fabe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7023899fb034854a56325cab379fabe" member_type: VOTER last_known_addr { host: "127.3.131.190" port: 39053 } } peers { permanent_uuid: "c4e36f6e2d6b4e2f9e95bdf4b99f2d08" member_type: VOTER last_known_addr { host: "127.3.131.189" port: 42239 } } peers { permanent_uuid: "8fb5e91e5845433999a13248ad4721ab" member_type: VOTER last_known_addr { host: "127.3.131.188" port: 35325 } }
I20260621 05:14:52.296671 26064 sys_catalog.cc:455] T 00000000000000000000000000000000 P f7023899fb034854a56325cab379fabe [sys.catalog]: SysCatalogTable state changed. Reason: New leader f7023899fb034854a56325cab379fabe. Latest consensus state: current_term: 1 leader_uuid: "f7023899fb034854a56325cab379fabe" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7023899fb034854a56325cab379fabe" member_type: VOTER last_known_addr { host: "127.3.131.190" port: 39053 } } peers { permanent_uuid: "c4e36f6e2d6b4e2f9e95bdf4b99f2d08" member_type: VOTER last_known_addr { host: "127.3.131.189" port: 42239 } } peers { permanent_uuid: "8fb5e91e5845433999a13248ad4721ab" member_type: VOTER last_known_addr { host: "127.3.131.188" port: 35325 } } }
I20260621 05:14:52.296803 26064 sys_catalog.cc:458] T 00000000000000000000000000000000 P f7023899fb034854a56325cab379fabe [sys.catalog]: This master's current role is: LEADER
I20260621 05:14:52.297035 26083 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260621 05:14:52.297282 26083 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260621 05:14:52.299643 26004 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 8fb5e91e5845433999a13248ad4721ab [term 1 FOLLOWER]: Refusing update from remote peer f7023899fb034854a56325cab379fabe: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 05:14:52.300061 26077 consensus_queue.cc:1035] T 00000000000000000000000000000000 P f7023899fb034854a56325cab379fabe [LEADER]: Connected to new peer: Peer: permanent_uuid: "8fb5e91e5845433999a13248ad4721ab" member_type: VOTER last_known_addr { host: "127.3.131.188" port: 35325 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 05:14:52.301736 26036 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8fb5e91e5845433999a13248ad4721ab [sys.catalog]: SysCatalogTable state changed. Reason: New leader f7023899fb034854a56325cab379fabe. Latest consensus state: current_term: 1 leader_uuid: "f7023899fb034854a56325cab379fabe" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7023899fb034854a56325cab379fabe" member_type: VOTER last_known_addr { host: "127.3.131.190" port: 39053 } } peers { permanent_uuid: "c4e36f6e2d6b4e2f9e95bdf4b99f2d08" member_type: VOTER last_known_addr { host: "127.3.131.189" port: 42239 } } peers { permanent_uuid: "8fb5e91e5845433999a13248ad4721ab" member_type: VOTER last_known_addr { host: "127.3.131.188" port: 35325 } } }
I20260621 05:14:52.301842 26036 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8fb5e91e5845433999a13248ad4721ab [sys.catalog]: This master's current role is: FOLLOWER
I20260621 05:14:52.302654 25929 raft_consensus.cc:1261] T 00000000000000000000000000000000 P c4e36f6e2d6b4e2f9e95bdf4b99f2d08 [term 1 FOLLOWER]: Refusing update from remote peer f7023899fb034854a56325cab379fabe: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 05:14:52.303040 26077 consensus_queue.cc:1035] T 00000000000000000000000000000000 P f7023899fb034854a56325cab379fabe [LEADER]: Connected to new peer: Peer: permanent_uuid: "c4e36f6e2d6b4e2f9e95bdf4b99f2d08" member_type: VOTER last_known_addr { host: "127.3.131.189" port: 42239 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 05:14:52.304529 26050 sys_catalog.cc:455] T 00000000000000000000000000000000 P c4e36f6e2d6b4e2f9e95bdf4b99f2d08 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f7023899fb034854a56325cab379fabe. Latest consensus state: current_term: 1 leader_uuid: "f7023899fb034854a56325cab379fabe" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7023899fb034854a56325cab379fabe" member_type: VOTER last_known_addr { host: "127.3.131.190" port: 39053 } } peers { permanent_uuid: "c4e36f6e2d6b4e2f9e95bdf4b99f2d08" member_type: VOTER last_known_addr { host: "127.3.131.189" port: 42239 } } peers { permanent_uuid: "8fb5e91e5845433999a13248ad4721ab" member_type: VOTER last_known_addr { host: "127.3.131.188" port: 35325 } } }
I20260621 05:14:52.304629 26050 sys_catalog.cc:458] T 00000000000000000000000000000000 P c4e36f6e2d6b4e2f9e95bdf4b99f2d08 [sys.catalog]: This master's current role is: FOLLOWER
I20260621 05:14:52.305769 26077 sys_catalog.cc:455] T 00000000000000000000000000000000 P f7023899fb034854a56325cab379fabe [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "f7023899fb034854a56325cab379fabe" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7023899fb034854a56325cab379fabe" member_type: VOTER last_known_addr { host: "127.3.131.190" port: 39053 } } peers { permanent_uuid: "c4e36f6e2d6b4e2f9e95bdf4b99f2d08" member_type: VOTER last_known_addr { host: "127.3.131.189" port: 42239 } } peers { permanent_uuid: "8fb5e91e5845433999a13248ad4721ab" member_type: VOTER last_known_addr { host: "127.3.131.188" port: 35325 } } }
I20260621 05:14:52.305867 26077 sys_catalog.cc:458] T 00000000000000000000000000000000 P f7023899fb034854a56325cab379fabe [sys.catalog]: This master's current role is: LEADER
I20260621 05:14:52.306318 26036 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8fb5e91e5845433999a13248ad4721ab [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "f7023899fb034854a56325cab379fabe" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7023899fb034854a56325cab379fabe" member_type: VOTER last_known_addr { host: "127.3.131.190" port: 39053 } } peers { permanent_uuid: "c4e36f6e2d6b4e2f9e95bdf4b99f2d08" member_type: VOTER last_known_addr { host: "127.3.131.189" port: 42239 } } peers { permanent_uuid: "8fb5e91e5845433999a13248ad4721ab" member_type: VOTER last_known_addr { host: "127.3.131.188" port: 35325 } } }
I20260621 05:14:52.306409 26036 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8fb5e91e5845433999a13248ad4721ab [sys.catalog]: This master's current role is: FOLLOWER
I20260621 05:14:52.306872 26084 mvcc.cc:205] Tried to move back new op lower bound from 7299149382856212480 to 7299149382845841408. Current Snapshot: MvccSnapshot[applied={T|T < 7299149382856212480}]
I20260621 05:14:52.307869 26085 mvcc.cc:205] Tried to move back new op lower bound from 7299149382856212480 to 7299149382845841408. Current Snapshot: MvccSnapshot[applied={T|T < 7299149382856212480}]
I20260621 05:14:52.307978 26064 sys_catalog.cc:455] T 00000000000000000000000000000000 P f7023899fb034854a56325cab379fabe [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "f7023899fb034854a56325cab379fabe" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7023899fb034854a56325cab379fabe" member_type: VOTER last_known_addr { host: "127.3.131.190" port: 39053 } } peers { permanent_uuid: "c4e36f6e2d6b4e2f9e95bdf4b99f2d08" member_type: VOTER last_known_addr { host: "127.3.131.189" port: 42239 } } peers { permanent_uuid: "8fb5e91e5845433999a13248ad4721ab" member_type: VOTER last_known_addr { host: "127.3.131.188" port: 35325 } } }
I20260621 05:14:52.308070 26064 sys_catalog.cc:458] T 00000000000000000000000000000000 P f7023899fb034854a56325cab379fabe [sys.catalog]: This master's current role is: LEADER
I20260621 05:14:52.308602 26050 sys_catalog.cc:455] T 00000000000000000000000000000000 P c4e36f6e2d6b4e2f9e95bdf4b99f2d08 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "f7023899fb034854a56325cab379fabe" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7023899fb034854a56325cab379fabe" member_type: VOTER last_known_addr { host: "127.3.131.190" port: 39053 } } peers { permanent_uuid: "c4e36f6e2d6b4e2f9e95bdf4b99f2d08" member_type: VOTER last_known_addr { host: "127.3.131.189" port: 42239 } } peers { permanent_uuid: "8fb5e91e5845433999a13248ad4721ab" member_type: VOTER last_known_addr { host: "127.3.131.188" port: 35325 } } }
I20260621 05:14:52.308691 26050 sys_catalog.cc:458] T 00000000000000000000000000000000 P c4e36f6e2d6b4e2f9e95bdf4b99f2d08 [sys.catalog]: This master's current role is: FOLLOWER
I20260621 05:14:52.309465 26089 mvcc.cc:205] Tried to move back new op lower bound from 7299149382856212480 to 7299149382845841408. Current Snapshot: MvccSnapshot[applied={T|T < 7299149382856212480}]
I20260621 05:14:52.310039 26083 catalog_manager.cc:1347] Generated new cluster ID: e00d5249c91a40e3970f7eef91d1965a
I20260621 05:14:52.310084 26083 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260621 05:14:52.319618 25959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:14:52.303596 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49616 (local address 127.3.131.189:42239)
0621 05:14:52.303641 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0621 05:14:52.303643 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:14:52.307072 (+ 3429us) server_negotiation.cc:407] Connection header received
0621 05:14:52.307203 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:14:52.307205 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:14:52.307216 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:14:52.307247 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:14:52.308071 (+ 824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:14:52.308825 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:14:52.309915 (+ 1090us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:14:52.310207 (+ 292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:14:52.310245 (+ 38us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:14:52.316475 (+ 6230us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:14:52.316477 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:14:52.316477 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:14:52.316485 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:14:52.318232 (+ 1747us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:14:52.318667 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:14:52.318668 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:14:52.318669 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:14:52.318723 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:14:52.318931 (+ 208us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:14:52.318931 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:14:52.318932 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:14:52.319063 (+ 131us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:14:52.319139 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0621 05:14:52.319412 (+ 273us) server_negotiation.cc:299] Negotiation successful
0621 05:14:52.319492 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260621 05:14:52.321980 26034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:14:52.304257 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53546 (local address 127.3.131.188:35325)
0621 05:14:52.304294 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0621 05:14:52.304296 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:14:52.313869 (+ 9573us) server_negotiation.cc:407] Connection header received
0621 05:14:52.313961 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:14:52.313962 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:14:52.313973 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:14:52.314001 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:14:52.315331 (+ 1330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:14:52.316094 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:14:52.317104 (+ 1010us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:14:52.317416 (+ 312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:14:52.317456 (+ 40us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:14:52.318881 (+ 1425us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:14:52.318882 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:14:52.318883 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:14:52.318890 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:14:52.320641 (+ 1751us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:14:52.321090 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:14:52.321092 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:14:52.321092 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:14:52.321152 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:14:52.321356 (+ 204us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:14:52.321357 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:14:52.321357 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:14:52.321491 (+ 134us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:14:52.321566 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0621 05:14:52.321792 (+ 226us) server_negotiation.cc:299] Negotiation successful
0621 05:14:52.321867 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260621 05:14:52.323135 25882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:14:52.303962 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38672 (local address 127.3.131.190:39053)
0621 05:14:52.304001 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0621 05:14:52.304003 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:14:52.311316 (+ 7313us) server_negotiation.cc:407] Connection header received
0621 05:14:52.311426 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:14:52.311427 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:14:52.311440 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:14:52.311473 (+ 33us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:14:52.312088 (+ 615us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:14:52.312787 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:14:52.314628 (+ 1841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:14:52.314894 (+ 266us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:14:52.314940 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:14:52.319808 (+ 4868us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:14:52.319810 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:14:52.319810 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:14:52.319820 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:14:52.321606 (+ 1786us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:14:52.322210 (+ 604us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:14:52.322211 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:14:52.322212 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:14:52.322273 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:14:52.322474 (+ 201us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:14:52.322475 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:14:52.322475 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:14:52.322616 (+ 141us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:14:52.322688 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0621 05:14:52.322951 (+ 263us) server_negotiation.cc:299] Negotiation successful
0621 05:14:52.323027 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260621 05:14:52.341338 26083 catalog_manager.cc:1370] Generated new certificate authority record
I20260621 05:14:52.342139 26083 catalog_manager.cc:1504] Loading token signing keys...
I20260621 05:14:52.366044 26083 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f7023899fb034854a56325cab379fabe: Generated new TSK 0
I20260621 05:14:52.366374 26083 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260621 05:14:52.389418 25959 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 05:14:52.387716 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49628 (local address 127.3.131.189:42239)
0621 05:14:52.387763 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0621 05:14:52.387765 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:14:52.388707 (+ 942us) server_negotiation.cc:407] Connection header received
0621 05:14:52.388816 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:14:52.388818 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:14:52.388829 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:14:52.388860 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:14:52.389347 (+ 487us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49628: BlockingRecv error: recv got EOF from 127.0.0.1:49628 (error 108)
Metrics: {"server-negotiator.queue_time_us":23}
W20260621 05:14:52.390411 25882 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 05:14:52.388138 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38674 (local address 127.3.131.190:39053)
0621 05:14:52.388190 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0621 05:14:52.388192 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:14:52.389848 (+ 1656us) server_negotiation.cc:407] Connection header received
0621 05:14:52.389949 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:14:52.389950 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:14:52.389962 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:14:52.389992 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:14:52.390346 (+ 354us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38674: BlockingRecv error: recv got EOF from 127.0.0.1:38674 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
W20260621 05:14:52.391341 26034 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 05:14:52.388449 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53554 (local address 127.3.131.188:35325)
0621 05:14:52.388493 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0621 05:14:52.388495 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:14:52.390759 (+ 2264us) server_negotiation.cc:407] Connection header received
0621 05:14:52.390892 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:14:52.390894 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:14:52.390905 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:14:52.390935 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:14:52.391281 (+ 346us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53554: BlockingRecv error: recv got EOF from 127.0.0.1:53554 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
W20260621 05:14:52.396348 25959 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 05:14:52.395508 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49644 (local address 127.3.131.189:42239)
0621 05:14:52.395550 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0621 05:14:52.395553 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0621 05:14:52.395767 (+ 214us) server_negotiation.cc:407] Connection header received
0621 05:14:52.395879 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:14:52.395880 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:14:52.395899 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:14:52.395928 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:14:52.396296 (+ 368us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49644: BlockingRecv error: recv got EOF from 127.0.0.1:49644 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
W20260621 05:14:52.399901 25959 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 05:14:52.398869 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49658 (local address 127.3.131.189:42239)
0621 05:14:52.398908 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0621 05:14:52.398910 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:14:52.399313 (+ 403us) server_negotiation.cc:407] Connection header received
0621 05:14:52.399431 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:14:52.399432 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:14:52.399443 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:14:52.399474 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:14:52.399847 (+ 373us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49658: BlockingRecv error: recv got EOF from 127.0.0.1:49658 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
W20260621 05:14:52.404284 25959 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 05:14:52.403310 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49666 (local address 127.3.131.189:42239)
0621 05:14:52.403354 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0621 05:14:52.403356 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:14:52.403657 (+ 301us) server_negotiation.cc:407] Connection header received
0621 05:14:52.403768 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:14:52.403769 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:14:52.403782 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:14:52.403812 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:14:52.404223 (+ 411us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49666: BlockingRecv error: recv got EOF from 127.0.0.1:49666 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
W20260621 05:14:52.411404 25959 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 05:14:52.409955 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49682 (local address 127.3.131.189:42239)
0621 05:14:52.410011 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0621 05:14:52.410013 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:14:52.410571 (+ 558us) server_negotiation.cc:407] Connection header received
0621 05:14:52.410708 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:14:52.410709 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:14:52.410725 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:14:52.410768 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:14:52.411337 (+ 569us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49682: BlockingRecv error: recv got EOF from 127.0.0.1:49682 (error 108)
Metrics: {"server-negotiator.queue_time_us":29}
W20260621 05:14:52.422788 25959 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 05:14:52.421146 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49692 (local address 127.3.131.189:42239)
0621 05:14:52.421237 (+ 91us) server_negotiation.cc:206] Beginning negotiation
0621 05:14:52.421240 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0621 05:14:52.421742 (+ 502us) server_negotiation.cc:407] Connection header received
0621 05:14:52.421941 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:14:52.421943 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:14:52.421959 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:14:52.421999 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:14:52.422705 (+ 706us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49692: BlockingRecv error: recv got EOF from 127.0.0.1:49692 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
W20260621 05:14:52.441439 25959 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 05:14:52.440642 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49708 (local address 127.3.131.189:42239)
0621 05:14:52.440690 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0621 05:14:52.440692 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:14:52.440823 (+ 131us) server_negotiation.cc:407] Connection header received
0621 05:14:52.440870 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:14:52.440871 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:14:52.440884 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:14:52.440981 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:14:52.441374 (+ 393us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49708: BlockingRecv error: recv got EOF from 127.0.0.1:49708 (error 108)
Metrics: {"server-negotiator.queue_time_us":23}
W20260621 05:14:52.475579 25959 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 05:14:52.474676 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49712 (local address 127.3.131.189:42239)
0621 05:14:52.474795 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0621 05:14:52.474798 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0621 05:14:52.474930 (+ 132us) server_negotiation.cc:407] Connection header received
0621 05:14:52.475039 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:14:52.475040 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:14:52.475052 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:14:52.475132 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:14:52.475520 (+ 388us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49712: BlockingRecv error: recv got EOF from 127.0.0.1:49712 (error 108)
Metrics: {"server-negotiator.queue_time_us":80}
W20260621 05:14:52.541841 25959 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 05:14:52.540938 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49728 (local address 127.3.131.189:42239)
0621 05:14:52.541032 (+ 94us) server_negotiation.cc:206] Beginning negotiation
0621 05:14:52.541035 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0621 05:14:52.541140 (+ 105us) server_negotiation.cc:407] Connection header received
0621 05:14:52.541270 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:14:52.541271 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:14:52.541283 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:14:52.541360 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:14:52.541793 (+ 433us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49728: BlockingRecv error: recv got EOF from 127.0.0.1:49728 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
W20260621 05:14:52.672276 25959 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 05:14:52.671346 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49732 (local address 127.3.131.189:42239)
0621 05:14:52.671421 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0621 05:14:52.671423 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:14:52.671507 (+ 84us) server_negotiation.cc:407] Connection header received
0621 05:14:52.671559 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:14:52.671560 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:14:52.671570 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:14:52.671640 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:14:52.672182 (+ 542us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49732: BlockingRecv error: recv got EOF from 127.0.0.1:49732 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
W20260621 05:14:52.930346 25959 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 05:14:52.929617 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49744 (local address 127.3.131.189:42239)
0621 05:14:52.929694 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0621 05:14:52.929696 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:14:52.929704 (+ 8us) server_negotiation.cc:407] Connection header received
0621 05:14:52.929731 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:14:52.929732 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:14:52.929741 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:14:52.929805 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:14:52.930262 (+ 457us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49744: BlockingRecv error: recv got EOF from 127.0.0.1:49744 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
I20260621 05:14:53.223788 26047 catalog_manager.cc:1259] Loaded cluster ID: e00d5249c91a40e3970f7eef91d1965a
I20260621 05:14:53.223868 26047 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 8fb5e91e5845433999a13248ad4721ab: loading cluster ID for follower catalog manager: success
I20260621 05:14:53.224810 26047 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 8fb5e91e5845433999a13248ad4721ab: acquiring CA information for follower catalog manager: success
I20260621 05:14:53.225176 26047 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 8fb5e91e5845433999a13248ad4721ab: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260621 05:14:53.250690 26061 catalog_manager.cc:1259] Loaded cluster ID: e00d5249c91a40e3970f7eef91d1965a
I20260621 05:14:53.250784 26061 catalog_manager.cc:1552] T 00000000000000000000000000000000 P c4e36f6e2d6b4e2f9e95bdf4b99f2d08: loading cluster ID for follower catalog manager: success
I20260621 05:14:53.251684 26061 catalog_manager.cc:1574] T 00000000000000000000000000000000 P c4e36f6e2d6b4e2f9e95bdf4b99f2d08: acquiring CA information for follower catalog manager: success
I20260621 05:14:53.252004 26061 catalog_manager.cc:1602] T 00000000000000000000000000000000 P c4e36f6e2d6b4e2f9e95bdf4b99f2d08: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260621 05:14:53.446456 26159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:14:53.443720 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49750 (local address 127.3.131.189:42239)
0621 05:14:53.443820 (+ 100us) server_negotiation.cc:206] Beginning negotiation
0621 05:14:53.443822 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:14:53.443864 (+ 42us) server_negotiation.cc:407] Connection header received
0621 05:14:53.443891 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:14:53.443891 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:14:53.443939 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:14:53.444009 (+ 70us) server_negotiation.cc:226] Negotiated authn=TOKEN
0621 05:14:53.444202 (+ 193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:14:53.444735 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:14:53.445571 (+ 836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:14:53.445774 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:14:53.445835 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:14:53.446093 (+ 258us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0621 05:14:53.446210 (+ 117us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0621 05:14:53.446273 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0621 05:14:53.446357 (+ 84us) server_negotiation.cc:299] Negotiation successful
0621 05:14:53.446379 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83,"thread_start_us":55,"threads_started":1}
I20260621 05:14:53.451464 26165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:14:53.448611 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53556 (local address 127.3.131.188:35325)
0621 05:14:53.448717 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0621 05:14:53.448719 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:14:53.448772 (+ 53us) server_negotiation.cc:407] Connection header received
0621 05:14:53.448855 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:14:53.448856 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:14:53.448906 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:14:53.449001 (+ 95us) server_negotiation.cc:226] Negotiated authn=TOKEN
0621 05:14:53.449146 (+ 145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:14:53.449732 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:14:53.450478 (+ 746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:14:53.450710 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:14:53.450768 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:14:53.450954 (+ 186us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0621 05:14:53.451079 (+ 125us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0621 05:14:53.451151 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0621 05:14:53.451372 (+ 221us) server_negotiation.cc:299] Negotiation successful
0621 05:14:53.451396 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81,"thread_start_us":47,"threads_started":1}
I20260621 05:14:53.456380 26159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:14:53.453351 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49756 (local address 127.3.131.189:42239)
0621 05:14:53.453401 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0621 05:14:53.453403 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:14:53.453560 (+ 157us) server_negotiation.cc:407] Connection header received
0621 05:14:53.453661 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:14:53.453662 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:14:53.453674 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:14:53.453751 (+ 77us) server_negotiation.cc:226] Negotiated authn=TOKEN
0621 05:14:53.453932 (+ 181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:14:53.454670 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:14:53.455608 (+ 938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:14:53.455817 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:14:53.455889 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:14:53.456046 (+ 157us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0621 05:14:53.456091 (+ 45us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0621 05:14:53.456161 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0621 05:14:53.456269 (+ 108us) server_negotiation.cc:299] Negotiation successful
0621 05:14:53.456299 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260621 05:14:53.456879 26165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:14:53.453583 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53564 (local address 127.3.131.188:35325)
0621 05:14:53.453644 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0621 05:14:53.453646 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:14:53.453746 (+ 100us) server_negotiation.cc:407] Connection header received
0621 05:14:53.453833 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:14:53.453834 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:14:53.453843 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:14:53.453906 (+ 63us) server_negotiation.cc:226] Negotiated authn=TOKEN
0621 05:14:53.454031 (+ 125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:14:53.454546 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:14:53.455627 (+ 1081us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:14:53.455925 (+ 298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:14:53.455962 (+ 37us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:14:53.456158 (+ 196us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0621 05:14:53.456190 (+ 32us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0621 05:14:53.456243 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0621 05:14:53.456781 (+ 538us) server_negotiation.cc:299] Negotiation successful
0621 05:14:53.456803 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260621 05:14:53.457105 26172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:14:53.453363 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38688 (local address 127.3.131.190:39053)
0621 05:14:53.453487 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0621 05:14:53.453489 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:14:53.453717 (+ 228us) server_negotiation.cc:407] Connection header received
0621 05:14:53.453796 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:14:53.453797 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:14:53.453850 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:14:53.453936 (+ 86us) server_negotiation.cc:226] Negotiated authn=TOKEN
0621 05:14:53.454157 (+ 221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:14:53.454873 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:14:53.455900 (+ 1027us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:14:53.456245 (+ 345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:14:53.456306 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:14:53.456547 (+ 241us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0621 05:14:53.456661 (+ 114us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0621 05:14:53.456740 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0621 05:14:53.456986 (+ 246us) server_negotiation.cc:299] Negotiation successful
0621 05:14:53.457016 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":102,"thread_start_us":68,"threads_started":1}
I20260621 05:14:53.459391 3598 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWZOx2_/build/release/bin/kudu with pid 25812
I20260621 05:14:53.465957 3598 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWZOx2_/build/release/bin/kudu with pid 25887
I20260621 05:14:53.471330 3598 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWZOx2_/build/release/bin/kudu with pid 25962
2026-06-21T05:14:53Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (2979 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782018616648578-3598-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 21 05:14:53 dist-test-slave-vrq1 krb5kdc[26176](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 21 05:14:53 dist-test-slave-vrq1 krb5kdc[26176](info): set up 2 sockets
Jun 21 05:14:53 dist-test-slave-vrq1 krb5kdc[26176](info): commencing operation
krb5kdc: starting...
W20260621 05:14:54.531859 3598 mini_kdc.cc:121] Time spent starting KDC: real 1.039s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782018616648578-3598-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782018616648578-3598-0/krb5kdc/test-user.keytab.
Jun 21 05:14:54 dist-test-slave-vrq1 krb5kdc[26176](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782018894, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-21T05:14:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-21T05:14:54Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.131.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.131.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.131.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782018616648578-3598-0/krb5kdc/kudu_127.3.131.190.keytab.
Entry for principal kudu/127.3.131.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782018616648578-3598-0/krb5kdc/kudu_127.3.131.190.keytab.
I20260621 05:14:54.663347 3598 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWZOx2_/build/release/bin/kudu
/tmp/dist-test-taskWZOx2_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782018616648578-3598-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782018616648578-3598-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782018616648578-3598-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782018616648578-3598-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.131.190:36663
--webserver_interface=127.3.131.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782018616648578-3598-0/krb5kdc/kudu_127.3.131.190.keytab
--principal=kudu/127.3.131.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.131.148:43003
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.131.190:36663,127.3.131.189:37625,127.3.131.188:32823
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782018616648578-3598-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782018616648578-3598-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782018616648578-3598-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 05:14:54.755402 26189 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260621 05:14:54.755717 26189 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 05:14:54.755760 26189 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 05:14:54.755810 26189 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 05:14:54.757159 26189 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 05:14:54.757237 26189 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 05:14:54.757266 26189 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 05:14:54.757298 26189 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 05:14:54.757332 26189 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 05:14:54.759128 26189 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.131.148:43003
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782018616648578-3598-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782018616648578-3598-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.131.190:36663,127.3.131.189:37625,127.3.131.188:32823
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782018616648578-3598-0/krb5kdc/kudu_127.3.131.190.keytab
--principal=kudu/127.3.131.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.131.190:36663
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782018616648578-3598-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.131.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26189
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782018616648578-3598-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 21 Jun 2026 04:16:16 UTC on ccycloud.quasar-xycaha.root.comops.site
build id 4457881
I20260621 05:14:54.759498 26189 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 05:14:54.759760 26189 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 05:14:54.765492 26195 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:14:54.765492 26194 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:14:54.765486 26197 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:14:54.765969 26189 server_base.cc:1034] running on GCE node
I20260621 05:14:54.766263 26189 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 05:14:54.766747 26189 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 05:14:54.767966 26189 hybrid_clock.cc:648] HybridClock initialized: now 1782018894767883 us; error 102 us; skew 500 ppm
Jun 21 05:14:54 dist-test-slave-vrq1 krb5kdc[26176](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782018894, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.131.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 05:14:54.776643 26189 init.cc:377] Logged in from keytab as kudu/127.3.131.190@KRBTEST.COM (short username kudu)
I20260621 05:14:54.777191 26189 webserver.cc:460] Webserver started at http://127.3.131.190:46039/ using document root <none> and password file <none>
I20260621 05:14:54.777408 26189 fs_manager.cc:359] Metadata directory not provided
I20260621 05:14:54.777452 26189 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 05:14:54.777550 26189 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 05:14:54.778616 26189 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782018616648578-3598-0/minicluster-data/master-0/data/instance:
uuid: "99b213827ec84131a97b66856d66ca8b"
format_stamp: "Formatted at 2026-06-21 05:14:54 on dist-test-slave-vrq1"
server_key: "083fe112c56ef5282415646703013be0"
server_key_iv: "df09e5c664d558044bccb5b316536aa1"
server_key_version: "encryptionkey@0"
I20260621 05:14:54.778934 26189 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782018616648578-3598-0/minicluster-data/master-0/wal/instance:
uuid: "99b213827ec84131a97b66856d66ca8b"
format_stamp: "Formatted at 2026-06-21 05:14:54 on dist-test-slave-vrq1"
server_key: "083fe112c56ef5282415646703013be0"
server_key_iv: "df09e5c664d558044bccb5b316536aa1"
server_key_version: "encryptionkey@0"
I20260621 05:14:54.780526 26189 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260621 05:14:54.781461 26204 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 05:14:54.781795 26189 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260621 05:14:54.781881 26189 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782018616648578-3598-0/minicluster-data/master-0/data,/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782018616648578-3598-0/minicluster-data/master-0/wal
uuid: "99b213827ec84131a97b66856d66ca8b"
format_stamp: "Formatted at 2026-06-21 05:14:54 on dist-test-slave-vrq1"
server_key: "083fe112c56ef5282415646703013be0"
server_key_iv: "df09e5c664d558044bccb5b316536aa1"
server_key_version: "encryptionkey@0"
I20260621 05:14:54.781985 26189 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782018616648578-3598-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782018616648578-3598-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782018616648578-3598-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 05:14:54.800565 26189 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 05:14:54.803784 26189 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 05:14:54.803916 26189 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 05:14:54.808897 26189 rpc_server.cc:307] RPC server started. Bound to: 127.3.131.190:36663
I20260621 05:14:54.808930 26256 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.131.190:36663 every 8 connection(s)
I20260621 05:14:54.809340 26189 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782018616648578-3598-0/minicluster-data/master-0/data/info.pb
I20260621 05:14:54.809810 3598 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWZOx2_/build/release/bin/kudu as pid 26189
I20260621 05:14:54.809895 3598 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782018616648578-3598-0/minicluster-data/master-0/wal/instance
I20260621 05:14:54.810109 3598 external_mini_cluster.cc:1428] Setting key 2215cb38ef44df020e3f4e4d292b11ca
I20260621 05:14:54.810734 26257 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 05:14:54.812597 26257 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.131.190" port: 36663 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.3.131.189@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.131.189@KRBTEST.COM" created.
Jun 21 05:14:54 dist-test-slave-vrq1 krb5kdc[26176](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782018894, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.131.190@KRBTEST.COM for kudu/127.3.131.190@KRBTEST.COM
I20260621 05:14:54.825083 26261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:14:54.813660 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36912 (local address 127.3.131.190:36663)
0621 05:14:54.813818 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0621 05:14:54.813823 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 05:14:54.813860 (+ 37us) server_negotiation.cc:407] Connection header received
0621 05:14:54.814514 (+ 654us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:14:54.814521 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:14:54.814541 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:14:54.814651 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:14:54.815355 (+ 704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:14:54.816325 (+ 970us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:14:54.817315 (+ 990us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:14:54.817697 (+ 382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:14:54.817869 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:14:54.821585 (+ 3716us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:14:54.821593 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:14:54.821594 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:14:54.821606 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:14:54.823427 (+ 1821us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:14:54.823854 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:14:54.823858 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:14:54.823858 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:14:54.823941 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:14:54.824171 (+ 230us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:14:54.824172 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:14:54.824173 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:14:54.824403 (+ 230us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:14:54.824507 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0621 05:14:54.824689 (+ 182us) server_negotiation.cc:299] Negotiation successful
0621 05:14:54.824827 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":127,"thread_start_us":70,"threads_started":1}
I20260621 05:14:54.825084 26260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:14:54.813433 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.131.190:36663 (local address 127.0.0.1:36912)
0621 05:14:54.813616 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0621 05:14:54.813704 (+ 88us) client_negotiation.cc:174] Beginning negotiation
0621 05:14:54.814317 (+ 613us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:14:54.814658 (+ 341us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:14:54.814660 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:14:54.815002 (+ 342us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:14:54.815239 (+ 237us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:14:54.815251 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:14:54.816448 (+ 1197us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:14:54.816449 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:14:54.817216 (+ 767us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:14:54.817218 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:14:54.817833 (+ 615us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:14:54.817834 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:14:54.817906 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:14:54.818567 (+ 661us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 05:14:54.818577 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:14:54.821462 (+ 2885us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 05:14:54.823535 (+ 2073us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:14:54.823542 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:14:54.823544 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:14:54.823748 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:14:54.824014 (+ 266us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:14:54.824015 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:14:54.824015 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:14:54.824103 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:14:54.824474 (+ 371us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 05:14:54.824479 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 05:14:54.824544 (+ 65us) client_negotiation.cc:769] Sending connection context
0621 05:14:54.824658 (+ 114us) client_negotiation.cc:240] Negotiation successful
0621 05:14:54.824891 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":139,"mutex_wait_us":84,"thread_start_us":85,"threads_started":1}
I20260621 05:14:54.825999 26257 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.131.189" port: 37625 } has no permanent_uuid. Determining permanent_uuid...
I20260621 05:14:54.826478 26260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:14:54.826332 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.131.189:37625 (local address 127.0.0.1:54688)
0621 05:14:54.826388 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0621 05:14:54.826442 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.131.189:37625: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260621 05:14:54.826639 26206 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.131.189:37625: connect: Connection refused (error 111)
I20260621 05:14:54.827155 26260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:14:54.827047 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.131.189:37625 (local address 127.0.0.1:54694)
0621 05:14:54.827101 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0621 05:14:54.827129 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.131.189:37625: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260621 05:14:54.827298 26257 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.131.189:37625: Network error: Client connection negotiation failed: client connection to 127.3.131.189:37625: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.131.189 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782018616648578-3598-0/krb5kdc/kudu_127.3.131.189.keytab.
Entry for principal kudu/127.3.131.189 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782018616648578-3598-0/krb5kdc/kudu_127.3.131.189.keytab.
I20260621 05:14:54.843386 3598 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWZOx2_/build/release/bin/kudu
/tmp/dist-test-taskWZOx2_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782018616648578-3598-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782018616648578-3598-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782018616648578-3598-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782018616648578-3598-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.131.189:37625
--webserver_interface=127.3.131.189
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782018616648578-3598-0/krb5kdc/kudu_127.3.131.189.keytab
--principal=kudu/127.3.131.189
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.131.148:43003
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.131.190:36663,127.3.131.189:37625,127.3.131.188:32823
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782018616648578-3598-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782018616648578-3598-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782018616648578-3598-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260621 05:14:54.876597 26257 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.131.189" port: 37625 } attempt: 1
I20260621 05:14:54.877223 26260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:14:54.877033 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.131.189:37625 (local address 127.0.0.1:54710)
0621 05:14:54.877127 (+ 94us) negotiation.cc:107] Waiting for socket to connect
0621 05:14:54.877181 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.131.189:37625: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260621 05:14:54.877745 26260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:14:54.877610 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.131.189:37625 (local address 127.0.0.1:54712)
0621 05:14:54.877688 (+ 78us) negotiation.cc:107] Waiting for socket to connect
0621 05:14:54.877722 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.131.189:37625: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
W20260621 05:14:54.877883 26257 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.131.189:37625: Network error: Client connection negotiation failed: client connection to 127.3.131.189:37625: connect: Connection refused (error 111)
W20260621 05:14:54.929721 26264 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260621 05:14:54.929983 26264 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 05:14:54.930013 26264 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 05:14:54.930056 26264 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 05:14:54.931314 26264 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 05:14:54.931387 26264 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 05:14:54.931411 26264 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 05:14:54.931432 26264 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 05:14:54.931459 26264 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 05:14:54.933069 26264 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.131.148:43003
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782018616648578-3598-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782018616648578-3598-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.3.131.190:36663,127.3.131.189:37625,127.3.131.188:32823
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782018616648578-3598-0/krb5kdc/kudu_127.3.131.189.keytab
--principal=kudu/127.3.131.189
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.131.189:37625
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782018616648578-3598-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.3.131.189
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26264
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782018616648578-3598-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 21 Jun 2026 04:16:16 UTC on ccycloud.quasar-xycaha.root.comops.site
build id 4457881
I20260621 05:14:54.933389 26264 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 05:14:54.933642 26264 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 05:14:54.939388 26272 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:14:54.939388 26269 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:14:54.939494 26264 server_base.cc:1034] running on GCE node
W20260621 05:14:54.939422 26270 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:14:54.939944 26264 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 05:14:54.940299 26264 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 05:14:54.941483 26264 hybrid_clock.cc:648] HybridClock initialized: now 1782018894941461 us; error 63 us; skew 500 ppm
Jun 21 05:14:54 dist-test-slave-vrq1 krb5kdc[26176](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782018894, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.131.189@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 05:14:54.944525 26264 init.cc:377] Logged in from keytab as kudu/127.3.131.189@KRBTEST.COM (short username kudu)
I20260621 05:14:54.944993 26264 webserver.cc:460] Webserver started at http://127.3.131.189:45295/ using document root <none> and password file <none>
I20260621 05:14:54.945245 26264 fs_manager.cc:359] Metadata directory not provided
I20260621 05:14:54.945320 26264 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 05:14:54.945482 26264 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 05:14:54.946055 26257 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.131.189" port: 37625 } attempt: 2
I20260621 05:14:54.946504 26260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:14:54.946349 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.131.189:37625 (local address 127.0.0.1:54724)
0621 05:14:54.946427 (+ 78us) negotiation.cc:107] Waiting for socket to connect
0621 05:14:54.946470 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.131.189:37625: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260621 05:14:54.946947 26260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:14:54.946854 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.131.189:37625 (local address 127.0.0.1:54728)
0621 05:14:54.946902 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0621 05:14:54.946929 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.131.189:37625: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260621 05:14:54.947075 26257 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.131.189:37625: Network error: Client connection negotiation failed: client connection to 127.3.131.189:37625: connect: Connection refused (error 111)
I20260621 05:14:54.946916 26264 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782018616648578-3598-0/minicluster-data/master-1/data/instance:
uuid: "081e3ee07d0040bba3a6c9dea95c4698"
format_stamp: "Formatted at 2026-06-21 05:14:54 on dist-test-slave-vrq1"
server_key: "5fac3e336a828fa00cd26519bbce26e8"
server_key_iv: "db1e3fada478348c37903e64e1b6b969"
server_key_version: "encryptionkey@0"
I20260621 05:14:54.947396 26264 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782018616648578-3598-0/minicluster-data/master-1/wal/instance:
uuid: "081e3ee07d0040bba3a6c9dea95c4698"
format_stamp: "Formatted at 2026-06-21 05:14:54 on dist-test-slave-vrq1"
server_key: "5fac3e336a828fa00cd26519bbce26e8"
server_key_iv: "db1e3fada478348c37903e64e1b6b969"
server_key_version: "encryptionkey@0"
I20260621 05:14:54.948874 26264 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260621 05:14:54.949764 26279 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 05:14:54.950084 26264 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260621 05:14:54.950176 26264 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782018616648578-3598-0/minicluster-data/master-1/data,/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782018616648578-3598-0/minicluster-data/master-1/wal
uuid: "081e3ee07d0040bba3a6c9dea95c4698"
format_stamp: "Formatted at 2026-06-21 05:14:54 on dist-test-slave-vrq1"
server_key: "5fac3e336a828fa00cd26519bbce26e8"
server_key_iv: "db1e3fada478348c37903e64e1b6b969"
server_key_version: "encryptionkey@0"
I20260621 05:14:54.950263 26264 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782018616648578-3598-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782018616648578-3598-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782018616648578-3598-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 05:14:54.967324 26264 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 05:14:54.969658 26264 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 05:14:54.969802 26264 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 05:14:54.974354 26264 rpc_server.cc:307] RPC server started. Bound to: 127.3.131.189:37625
I20260621 05:14:54.974395 26331 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.131.189:37625 every 8 connection(s)
I20260621 05:14:54.974800 26264 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782018616648578-3598-0/minicluster-data/master-1/data/info.pb
I20260621 05:14:54.975993 26332 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 05:14:54.977857 26332 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.131.190" port: 36663 } has no permanent_uuid. Determining permanent_uuid...
I20260621 05:14:54.979203 3598 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWZOx2_/build/release/bin/kudu as pid 26264
I20260621 05:14:54.979310 3598 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782018616648578-3598-0/minicluster-data/master-1/wal/instance
I20260621 05:14:54.979513 3598 external_mini_cluster.cc:1428] Setting key 7586141940a8a58a26f84f3391e40cc2
Jun 21 05:14:54 dist-test-slave-vrq1 krb5kdc[26176](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782018894, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.131.189@KRBTEST.COM for kudu/127.3.131.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.3.131.188@KRBTEST.COM; defaulting to no policy
I20260621 05:14:54.988343 26261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:14:54.978832 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36916 (local address 127.3.131.190:36663)
0621 05:14:54.978927 (+ 95us) server_negotiation.cc:206] Beginning negotiation
0621 05:14:54.978930 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0621 05:14:54.979132 (+ 202us) server_negotiation.cc:407] Connection header received
0621 05:14:54.979879 (+ 747us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:14:54.979880 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:14:54.979891 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:14:54.979951 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:14:54.980905 (+ 954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:14:54.981410 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:14:54.982543 (+ 1133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:14:54.982763 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:14:54.982855 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:14:54.985257 (+ 2402us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:14:54.985258 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:14:54.985259 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:14:54.985267 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:14:54.986706 (+ 1439us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:14:54.987271 (+ 565us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:14:54.987273 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:14:54.987273 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:14:54.987336 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:14:54.987613 (+ 277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:14:54.987614 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:14:54.987614 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:14:54.987741 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:14:54.987820 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0621 05:14:54.988154 (+ 334us) server_negotiation.cc:299] Negotiation successful
0621 05:14:54.988247 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260621 05:14:54.988533 26334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:14:54.978705 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.131.190:36663 (local address 127.0.0.1:36916)
0621 05:14:54.978946 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0621 05:14:54.978999 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0621 05:14:54.979743 (+ 744us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:14:54.980049 (+ 306us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:14:54.980054 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:14:54.980456 (+ 402us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:14:54.980762 (+ 306us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:14:54.980769 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:14:54.981611 (+ 842us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:14:54.981613 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:14:54.982442 (+ 829us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:14:54.982444 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:14:54.982886 (+ 442us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:14:54.982887 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:14:54.983036 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:14:54.983668 (+ 632us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 05:14:54.983683 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:14:54.985109 (+ 1426us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 05:14:54.986825 (+ 1716us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:14:54.986834 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:14:54.986836 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:14:54.987142 (+ 306us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:14:54.987406 (+ 264us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:14:54.987407 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:14:54.987408 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:14:54.987535 (+ 127us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:14:54.987810 (+ 275us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 05:14:54.987817 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 05:14:54.987994 (+ 177us) client_negotiation.cc:769] Sending connection context
0621 05:14:54.988118 (+ 124us) client_negotiation.cc:240] Negotiation successful
0621 05:14:54.988285 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":161,"thread_start_us":97,"threads_started":1}
I20260621 05:14:54.989243 26332 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.131.189" port: 37625 } has no permanent_uuid. Determining permanent_uuid...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.131.188@KRBTEST.COM" created.
Jun 21 05:14:54 dist-test-slave-vrq1 krb5kdc[26176](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782018894, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.131.189@KRBTEST.COM for kudu/127.3.131.189@KRBTEST.COM
I20260621 05:14:54.997396 26334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:14:54.989545 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.131.189:37625 (local address 127.0.0.1:54742)
0621 05:14:54.989697 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0621 05:14:54.989711 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0621 05:14:54.989762 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:14:54.990089 (+ 327us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:14:54.990090 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:14:54.990215 (+ 125us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:14:54.990268 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:14:54.990270 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:14:54.991205 (+ 935us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:14:54.991206 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:14:54.991859 (+ 653us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:14:54.991861 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:14:54.992381 (+ 520us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:14:54.992381 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:14:54.992446 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:14:54.992957 (+ 511us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 05:14:54.992963 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:14:54.994403 (+ 1440us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 05:14:54.996280 (+ 1877us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:14:54.996281 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:14:54.996282 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:14:54.996448 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:14:54.996669 (+ 221us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:14:54.996670 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:14:54.996671 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:14:54.996726 (+ 55us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:14:54.997034 (+ 308us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 05:14:54.997035 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 05:14:54.997077 (+ 42us) client_negotiation.cc:769] Sending connection context
0621 05:14:54.997149 (+ 72us) client_negotiation.cc:240] Negotiation successful
0621 05:14:54.997255 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":133}
I20260621 05:14:54.997416 26336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:14:54.989655 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54742 (local address 127.3.131.189:37625)
0621 05:14:54.989861 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0621 05:14:54.989867 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 05:14:54.989893 (+ 26us) server_negotiation.cc:407] Connection header received
0621 05:14:54.989917 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:14:54.989920 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:14:54.989935 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:14:54.990020 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:14:54.990375 (+ 355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:14:54.991080 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:14:54.991953 (+ 873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:14:54.992292 (+ 339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:14:54.992381 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:14:54.994490 (+ 2109us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:14:54.994507 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:14:54.994508 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:14:54.994515 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:14:54.996170 (+ 1655us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:14:54.996540 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:14:54.996544 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:14:54.996544 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:14:54.996605 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:14:54.996794 (+ 189us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:14:54.996795 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:14:54.996796 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:14:54.996960 (+ 164us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:14:54.997061 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0621 05:14:54.997182 (+ 121us) server_negotiation.cc:299] Negotiation successful
0621 05:14:54.997309 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":60,"server-negotiator.queue_time_us":172,"thread_start_us":121,"threads_started":1}
I20260621 05:14:54.998196 26332 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.131.188" port: 32823 } has no permanent_uuid. Determining permanent_uuid...
I20260621 05:14:54.998651 26334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:14:54.998466 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.131.188:32823 (local address 127.0.0.1:41656)
0621 05:14:54.998568 (+ 102us) negotiation.cc:107] Waiting for socket to connect
0621 05:14:54.998623 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.131.188:32823: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83}
W20260621 05:14:54.998795 26282 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.131.188:32823: connect: Connection refused (error 111)
I20260621 05:14:54.999462 26334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:14:54.999336 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.131.188:32823 (local address 127.0.0.1:41668)
0621 05:14:54.999394 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0621 05:14:54.999424 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.131.188:32823: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260621 05:14:54.999729 26332 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.131.188:32823: Network error: Client connection negotiation failed: client connection to 127.3.131.188:32823: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.131.188 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782018616648578-3598-0/krb5kdc/kudu_127.3.131.188.keytab.
Entry for principal kudu/127.3.131.188 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782018616648578-3598-0/krb5kdc/kudu_127.3.131.188.keytab.
I20260621 05:14:55.006914 3598 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWZOx2_/build/release/bin/kudu
/tmp/dist-test-taskWZOx2_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782018616648578-3598-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782018616648578-3598-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782018616648578-3598-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782018616648578-3598-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.131.188:32823
--webserver_interface=127.3.131.188
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782018616648578-3598-0/krb5kdc/kudu_127.3.131.188.keytab
--principal=kudu/127.3.131.188
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.131.148:43003
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.131.190:36663,127.3.131.189:37625,127.3.131.188:32823
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782018616648578-3598-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782018616648578-3598-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782018616648578-3598-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260621 05:14:55.038297 26257 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.131.189" port: 37625 } attempt: 3
Jun 21 05:14:55 dist-test-slave-vrq1 krb5kdc[26176](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782018894, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.131.190@KRBTEST.COM for kudu/127.3.131.189@KRBTEST.COM
I20260621 05:14:55.049212 26332 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.131.188" port: 32823 } attempt: 1
I20260621 05:14:55.049646 26334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:14:55.049510 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.131.188:32823 (local address 127.0.0.1:41674)
0621 05:14:55.049573 (+ 63us) negotiation.cc:107] Waiting for socket to connect
0621 05:14:55.049606 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.131.188:32823: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
I20260621 05:14:55.049944 26334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:14:55.049845 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.131.188:32823 (local address 127.0.0.1:41680)
0621 05:14:55.049892 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0621 05:14:55.049917 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.131.188:32823: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260621 05:14:55.050024 26332 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.131.188:32823: Network error: Client connection negotiation failed: client connection to 127.3.131.188:32823: connect: Connection refused (error 111)
I20260621 05:14:55.051397 26336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:14:55.038859 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54756 (local address 127.3.131.189:37625)
0621 05:14:55.038906 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0621 05:14:55.038908 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:14:55.038920 (+ 12us) server_negotiation.cc:407] Connection header received
0621 05:14:55.039016 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:14:55.039017 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:14:55.039033 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:14:55.039093 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:14:55.039621 (+ 528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:14:55.042597 (+ 2976us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:14:55.043892 (+ 1295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:14:55.044330 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:14:55.044434 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:14:55.047115 (+ 2681us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:14:55.047117 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:14:55.047117 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:14:55.047128 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:14:55.050111 (+ 2983us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:14:55.050534 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:14:55.050535 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:14:55.050536 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:14:55.050606 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:14:55.050845 (+ 239us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:14:55.050846 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:14:55.050847 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:14:55.050982 (+ 135us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:14:55.051049 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0621 05:14:55.051254 (+ 205us) server_negotiation.cc:299] Negotiation successful
0621 05:14:55.051341 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260621 05:14:55.051420 26260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:14:55.038800 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.131.189:37625 (local address 127.0.0.1:54756)
0621 05:14:55.038867 (+ 67us) negotiation.cc:107] Waiting for socket to connect
0621 05:14:55.038881 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0621 05:14:55.038927 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:14:55.039150 (+ 223us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:14:55.039151 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:14:55.039441 (+ 290us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:14:55.039527 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:14:55.039530 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:14:55.042739 (+ 3209us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:14:55.042741 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:14:55.043769 (+ 1028us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:14:55.043773 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:14:55.044473 (+ 700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:14:55.044474 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:14:55.044591 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:14:55.045182 (+ 591us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 05:14:55.045188 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:14:55.046947 (+ 1759us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 05:14:55.050182 (+ 3235us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:14:55.050184 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:14:55.050184 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:14:55.050419 (+ 235us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:14:55.050687 (+ 268us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:14:55.050687 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:14:55.050688 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:14:55.050757 (+ 69us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:14:55.051054 (+ 297us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 05:14:55.051055 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 05:14:55.051117 (+ 62us) client_negotiation.cc:769] Sending connection context
0621 05:14:55.051213 (+ 96us) client_negotiation.cc:240] Negotiation successful
0621 05:14:55.051325 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":44}
I20260621 05:14:55.052487 26257 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.131.188" port: 32823 } has no permanent_uuid. Determining permanent_uuid...
I20260621 05:14:55.052877 26260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:14:55.052773 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.131.188:32823 (local address 127.0.0.1:41688)
0621 05:14:55.052825 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0621 05:14:55.052857 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.131.188:32823: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
I20260621 05:14:55.053232 26260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:14:55.053119 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.131.188:32823 (local address 127.0.0.1:41690)
0621 05:14:55.053147 (+ 28us) negotiation.cc:107] Waiting for socket to connect
0621 05:14:55.053208 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.131.188:32823: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":16}
W20260621 05:14:55.053339 26257 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.131.188:32823: Network error: Client connection negotiation failed: client connection to 127.3.131.188:32823: connect: Connection refused (error 111)
I20260621 05:14:55.084548 26257 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.131.188" port: 32823 } attempt: 1
I20260621 05:14:55.085146 26260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:14:55.084953 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.131.188:32823 (local address 127.0.0.1:41698)
0621 05:14:55.085053 (+ 100us) negotiation.cc:107] Waiting for socket to connect
0621 05:14:55.085101 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.131.188:32823: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
I20260621 05:14:55.085610 26260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:14:55.085474 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.131.188:32823 (local address 127.0.0.1:41704)
0621 05:14:55.085549 (+ 75us) negotiation.cc:107] Waiting for socket to connect
0621 05:14:55.085584 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.131.188:32823: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260621 05:14:55.085757 26257 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.131.188:32823: Network error: Client connection negotiation failed: client connection to 127.3.131.188:32823: connect: Connection refused (error 111)
W20260621 05:14:55.097543 26339 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260621 05:14:55.097833 26339 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 05:14:55.097873 26339 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 05:14:55.097915 26339 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 05:14:55.099138 26339 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 05:14:55.099277 26339 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 05:14:55.099308 26339 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 05:14:55.099335 26339 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 05:14:55.099364 26339 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 05:14:55.100976 26339 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.131.148:43003
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782018616648578-3598-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782018616648578-3598-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.3.131.190:36663,127.3.131.189:37625,127.3.131.188:32823
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782018616648578-3598-0/krb5kdc/kudu_127.3.131.188.keytab
--principal=kudu/127.3.131.188
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.131.188:32823
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782018616648578-3598-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.3.131.188
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26339
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782018616648578-3598-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 21 Jun 2026 04:16:16 UTC on ccycloud.quasar-xycaha.root.comops.site
build id 4457881
I20260621 05:14:55.101287 26339 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 05:14:55.101531 26339 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 05:14:55.107000 26344 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:14:55.107012 26347 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:14:55.107092 26339 server_base.cc:1034] running on GCE node
W20260621 05:14:55.107012 26345 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:14:55.107455 26339 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 05:14:55.107736 26339 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 05:14:55.108948 26339 hybrid_clock.cc:648] HybridClock initialized: now 1782018895108931 us; error 29 us; skew 500 ppm
Jun 21 05:14:55 dist-test-slave-vrq1 krb5kdc[26176](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782018895, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.131.188@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 05:14:55.111830 26339 init.cc:377] Logged in from keytab as kudu/127.3.131.188@KRBTEST.COM (short username kudu)
I20260621 05:14:55.112414 26339 webserver.cc:460] Webserver started at http://127.3.131.188:34971/ using document root <none> and password file <none>
I20260621 05:14:55.112677 26339 fs_manager.cc:359] Metadata directory not provided
I20260621 05:14:55.112752 26339 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 05:14:55.112923 26339 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 05:14:55.114073 26339 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782018616648578-3598-0/minicluster-data/master-2/data/instance:
uuid: "6064b320c9fc476bb74da93a4021d44f"
format_stamp: "Formatted at 2026-06-21 05:14:55 on dist-test-slave-vrq1"
server_key: "a16a3988b4e6db2686e8e1719423c6e4"
server_key_iv: "05853260fdaf745ceae5f55f7acb8973"
server_key_version: "encryptionkey@0"
I20260621 05:14:55.114475 26339 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782018616648578-3598-0/minicluster-data/master-2/wal/instance:
uuid: "6064b320c9fc476bb74da93a4021d44f"
format_stamp: "Formatted at 2026-06-21 05:14:55 on dist-test-slave-vrq1"
server_key: "a16a3988b4e6db2686e8e1719423c6e4"
server_key_iv: "05853260fdaf745ceae5f55f7acb8973"
server_key_version: "encryptionkey@0"
I20260621 05:14:55.116042 26339 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260621 05:14:55.117010 26354 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 05:14:55.117234 26339 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260621 05:14:55.117349 26339 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782018616648578-3598-0/minicluster-data/master-2/data,/tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782018616648578-3598-0/minicluster-data/master-2/wal
uuid: "6064b320c9fc476bb74da93a4021d44f"
format_stamp: "Formatted at 2026-06-21 05:14:55 on dist-test-slave-vrq1"
server_key: "a16a3988b4e6db2686e8e1719423c6e4"
server_key_iv: "05853260fdaf745ceae5f55f7acb8973"
server_key_version: "encryptionkey@0"
I20260621 05:14:55.117434 26339 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782018616648578-3598-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782018616648578-3598-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782018616648578-3598-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 05:14:55.118170 26332 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.131.188" port: 32823 } attempt: 2
I20260621 05:14:55.118665 26334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:14:55.118516 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.131.188:32823 (local address 127.0.0.1:41712)
0621 05:14:55.118581 (+ 65us) negotiation.cc:107] Waiting for socket to connect
0621 05:14:55.118624 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.131.188:32823: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
I20260621 05:14:55.119056 26334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:14:55.118969 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.131.188:32823 (local address 127.0.0.1:41714)
0621 05:14:55.119011 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0621 05:14:55.119037 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.131.188:32823: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260621 05:14:55.119155 26332 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.131.188:32823: Network error: Client connection negotiation failed: client connection to 127.3.131.188:32823: connect: Connection refused (error 111)
I20260621 05:14:55.136168 26339 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 05:14:55.138718 26339 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 05:14:55.138846 26339 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 05:14:55.143625 26339 rpc_server.cc:307] RPC server started. Bound to: 127.3.131.188:32823
I20260621 05:14:55.143692 26406 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.131.188:32823 every 8 connection(s)
I20260621 05:14:55.144053 26339 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782018616648578-3598-0/minicluster-data/master-2/data/info.pb
I20260621 05:14:55.145053 26407 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 05:14:55.146787 26407 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.131.190" port: 36663 } has no permanent_uuid. Determining permanent_uuid...
I20260621 05:14:55.153218 3598 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWZOx2_/build/release/bin/kudu as pid 26339
I20260621 05:14:55.153354 3598 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWZOx2_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782018616648578-3598-0/minicluster-data/master-2/wal/instance
I20260621 05:14:55.153600 3598 external_mini_cluster.cc:1428] Setting key 8b4013a29eccf10cacc2cb5bbe09ecce
Jun 21 05:14:55 dist-test-slave-vrq1 krb5kdc[26176](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782018895, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.131.188@KRBTEST.COM for kudu/127.3.131.190@KRBTEST.COM
I20260621 05:14:55.158552 26261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:14:55.147661 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36926 (local address 127.3.131.190:36663)
0621 05:14:55.147765 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0621 05:14:55.147768 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0621 05:14:55.147953 (+ 185us) server_negotiation.cc:407] Connection header received
0621 05:14:55.148552 (+ 599us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:14:55.148553 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:14:55.148564 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:14:55.148637 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:14:55.149427 (+ 790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:14:55.149916 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:14:55.151152 (+ 1236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:14:55.151447 (+ 295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:14:55.151648 (+ 201us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:14:55.154937 (+ 3289us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:14:55.154939 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:14:55.154940 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:14:55.154947 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:14:55.156797 (+ 1850us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:14:55.157376 (+ 579us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:14:55.157377 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:14:55.157378 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:14:55.157442 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:14:55.157795 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:14:55.157796 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:14:55.157797 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:14:55.157946 (+ 149us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:14:55.158022 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0621 05:14:55.158352 (+ 330us) server_negotiation.cc:299] Negotiation successful
0621 05:14:55.158429 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260621 05:14:55.158775 26409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:14:55.147554 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.131.190:36663 (local address 127.0.0.1:36926)
0621 05:14:55.147776 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0621 05:14:55.147818 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0621 05:14:55.148422 (+ 604us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:14:55.148772 (+ 350us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:14:55.148777 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:14:55.149045 (+ 268us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:14:55.149273 (+ 228us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:14:55.149278 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:14:55.150186 (+ 908us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:14:55.150187 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:14:55.151042 (+ 855us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:14:55.151044 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:14:55.151673 (+ 629us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:14:55.151674 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:14:55.151799 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:14:55.152593 (+ 794us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 05:14:55.152603 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:14:55.154822 (+ 2219us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 05:14:55.156886 (+ 2064us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:14:55.156893 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:14:55.156896 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:14:55.157274 (+ 378us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:14:55.157586 (+ 312us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:14:55.157586 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:14:55.157587 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:14:55.157726 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:14:55.158031 (+ 305us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 05:14:55.158036 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 05:14:55.158168 (+ 132us) client_negotiation.cc:769] Sending connection context
0621 05:14:55.158306 (+ 138us) client_negotiation.cc:240] Negotiation successful
0621 05:14:55.158493 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":170,"thread_start_us":98,"threads_started":1}
I20260621 05:14:55.159936 26407 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.131.189" port: 37625 } has no permanent_uuid. Determining permanent_uuid...
Jun 21 05:14:55 dist-test-slave-vrq1 krb5kdc[26176](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782018894, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.131.188@KRBTEST.COM
I20260621 05:14:55.160925 26257 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.131.188" port: 32823 } attempt: 2
Jun 21 05:14:55 dist-test-slave-vrq1 krb5kdc[26176](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782018895, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.131.188@KRBTEST.COM for kudu/127.3.131.189@KRBTEST.COM
I20260621 05:14:55.165149 26411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:14:55.154109 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41730 (local address 127.3.131.188:32823)
0621 05:14:55.154282 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0621 05:14:55.154287 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 05:14:55.154384 (+ 97us) server_negotiation.cc:407] Connection header received
0621 05:14:55.154419 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:14:55.154421 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:14:55.154442 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:14:55.154526 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:14:55.155350 (+ 824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:14:55.156496 (+ 1146us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:14:55.157648 (+ 1152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:14:55.158148 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:14:55.158204 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:14:55.161171 (+ 2967us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:14:55.161178 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:14:55.161179 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:14:55.161187 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:14:55.163383 (+ 2196us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:14:55.163801 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:14:55.163805 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:14:55.163806 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:14:55.163868 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:14:55.164100 (+ 232us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:14:55.164101 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:14:55.164101 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:14:55.164297 (+ 196us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:14:55.164499 (+ 202us) server_negotiation.cc:1035] Waiting for connection context
0621 05:14:55.164545 (+ 46us) server_negotiation.cc:299] Negotiation successful
0621 05:14:55.165027 (+ 482us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":132,"thread_start_us":103,"threads_started":1}
Jun 21 05:14:55 dist-test-slave-vrq1 krb5kdc[26176](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782018894, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.131.190@KRBTEST.COM for kudu/127.3.131.188@KRBTEST.COM
I20260621 05:14:55.168197 26336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:14:55.160340 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54766 (local address 127.3.131.189:37625)
0621 05:14:55.160392 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0621 05:14:55.160394 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:14:55.160450 (+ 56us) server_negotiation.cc:407] Connection header received
0621 05:14:55.160538 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:14:55.160539 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:14:55.160548 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:14:55.160589 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:14:55.160892 (+ 303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:14:55.161613 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:14:55.162383 (+ 770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:14:55.162648 (+ 265us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:14:55.162716 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:14:55.165023 (+ 2307us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:14:55.165025 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:14:55.165025 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:14:55.165035 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:14:55.166919 (+ 1884us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:14:55.167351 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:14:55.167352 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:14:55.167353 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:14:55.167420 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:14:55.167645 (+ 225us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:14:55.167646 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:14:55.167646 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:14:55.167767 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:14:55.167844 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0621 05:14:55.168001 (+ 157us) server_negotiation.cc:299] Negotiation successful
0621 05:14:55.168098 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260621 05:14:55.168206 26409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:14:55.160256 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.131.189:37625 (local address 127.0.0.1:54766)
0621 05:14:55.160401 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0621 05:14:55.160412 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0621 05:14:55.160468 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:14:55.160602 (+ 134us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:14:55.160603 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:14:55.160759 (+ 156us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:14:55.160814 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:14:55.160816 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:14:55.161698 (+ 882us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:14:55.161699 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:14:55.162294 (+ 595us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:14:55.162296 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:14:55.162729 (+ 433us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:14:55.162730 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:14:55.162811 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:14:55.163250 (+ 439us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 05:14:55.163256 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:14:55.164915 (+ 1659us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 05:14:55.167005 (+ 2090us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:14:55.167006 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:14:55.167006 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:14:55.167266 (+ 260us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:14:55.167503 (+ 237us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:14:55.167503 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:14:55.167504 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:14:55.167564 (+ 60us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:14:55.167835 (+ 271us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 05:14:55.167836 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 05:14:55.167882 (+ 46us) client_negotiation.cc:769] Sending connection context
0621 05:14:55.167972 (+ 90us) client_negotiation.cc:240] Negotiation successful
0621 05:14:55.168118 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123}
I20260621 05:14:55.168764 26407 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.131.188" port: 32823 } has no permanent_uuid. Determining permanent_uuid...
I20260621 05:14:55.170436 26260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:14:55.161182 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.131.188:32823 (local address 127.0.0.1:41738)
0621 05:14:55.161228 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0621 05:14:55.161239 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0621 05:14:55.161275 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:14:55.161896 (+ 621us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:14:55.161897 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:14:55.162115 (+ 218us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:14:55.162484 (+ 369us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:14:55.162487 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:14:55.163376 (+ 889us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:14:55.163377 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:14:55.164176 (+ 799us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:14:55.164178 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:14:55.164673 (+ 495us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:14:55.164674 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:14:55.164739 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:14:55.165143 (+ 404us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 05:14:55.165148 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:14:55.166716 (+ 1568us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 05:14:55.169354 (+ 2638us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:14:55.169355 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:14:55.169355 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:14:55.169592 (+ 237us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:14:55.169802 (+ 210us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:14:55.169803 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:14:55.169803 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:14:55.169863 (+ 60us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:14:55.170143 (+ 280us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 05:14:55.170144 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 05:14:55.170191 (+ 47us) client_negotiation.cc:769] Sending connection context
0621 05:14:55.170237 (+ 46us) client_negotiation.cc:240] Negotiation successful
0621 05:14:55.170335 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":24}
I20260621 05:14:55.171028 26412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:14:55.161270 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41738 (local address 127.3.131.188:32823)
0621 05:14:55.161764 (+ 494us) server_negotiation.cc:206] Beginning negotiation
0621 05:14:55.161766 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:14:55.161772 (+ 6us) server_negotiation.cc:407] Connection header received
0621 05:14:55.161789 (+ 17us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:14:55.161789 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:14:55.161800 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:14:55.161865 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:14:55.162573 (+ 708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:14:55.163264 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:14:55.164253 (+ 989us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:14:55.164563 (+ 310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:14:55.164909 (+ 346us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:14:55.166808 (+ 1899us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:14:55.166809 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:14:55.166810 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:14:55.166818 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:14:55.169241 (+ 2423us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:14:55.169659 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:14:55.169661 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:14:55.169661 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:14:55.169723 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:14:55.169930 (+ 207us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:14:55.169931 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:14:55.169931 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:14:55.170063 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:14:55.170754 (+ 691us) server_negotiation.cc:1035] Waiting for connection context
0621 05:14:55.170797 (+ 43us) server_negotiation.cc:299] Negotiation successful
0621 05:14:55.170877 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":471,"thread_start_us":129,"threads_started":1}
Jun 21 05:14:55 dist-test-slave-vrq1 krb5kdc[26176](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782018894, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.131.190@KRBTEST.COM
I20260621 05:14:55.173162 26257 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 99b213827ec84131a97b66856d66ca8b: Bootstrap starting.
Jun 21 05:14:55 dist-test-slave-vrq1 krb5kdc[26176](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782018895, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.131.188@KRBTEST.COM for kudu/127.3.131.188@KRBTEST.COM
I20260621 05:14:55.174053 26257 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 99b213827ec84131a97b66856d66ca8b: Neither blocks nor log segments found. Creating new log.
I20260621 05:14:55.174420 26257 log.cc:824] T 00000000000000000000000000000000 P 99b213827ec84131a97b66856d66ca8b: Log is configured to *not* fsync() on all Append() calls
I20260621 05:14:55.175675 26257 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 99b213827ec84131a97b66856d66ca8b: No bootstrap required, opened a new log
I20260621 05:14:55.176533 26261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:14:55.166105 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36932 (local address 127.3.131.190:36663)
0621 05:14:55.166287 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0621 05:14:55.166289 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:14:55.166298 (+ 9us) server_negotiation.cc:407] Connection header received
0621 05:14:55.166320 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:14:55.166321 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:14:55.166332 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:14:55.166408 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:14:55.167248 (+ 840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:14:55.168049 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:14:55.169740 (+ 1691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:14:55.170040 (+ 300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:14:55.170111 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:14:55.173095 (+ 2984us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:14:55.173096 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:14:55.173097 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:14:55.173105 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:14:55.174759 (+ 1654us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:14:55.175287 (+ 528us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:14:55.175288 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:14:55.175289 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:14:55.175352 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:14:55.175680 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:14:55.175681 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:14:55.175682 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:14:55.175853 (+ 171us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:14:55.175973 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0621 05:14:55.176197 (+ 224us) server_negotiation.cc:299] Negotiation successful
0621 05:14:55.176296 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":158}
I20260621 05:14:55.177213 26409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:14:55.169058 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.131.188:32823 (local address 127.0.0.1:41744)
0621 05:14:55.169089 (+ 31us) negotiation.cc:107] Waiting for socket to connect
0621 05:14:55.169099 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0621 05:14:55.169135 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:14:55.169282 (+ 147us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:14:55.169282 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:14:55.169438 (+ 156us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:14:55.169476 (+ 38us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:14:55.169478 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:14:55.170184 (+ 706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:14:55.170185 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:14:55.171081 (+ 896us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:14:55.171084 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:14:55.171559 (+ 475us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:14:55.171560 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:14:55.171605 (+ 45us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:14:55.172001 (+ 396us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 05:14:55.172006 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:14:55.173725 (+ 1719us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 05:14:55.176010 (+ 2285us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:14:55.176011 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:14:55.176012 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:14:55.176277 (+ 265us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:14:55.176522 (+ 245us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:14:55.176523 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:14:55.176524 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:14:55.176596 (+ 72us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:14:55.176931 (+ 335us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 05:14:55.176931 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 05:14:55.176971 (+ 40us) client_negotiation.cc:769] Sending connection context
0621 05:14:55.177014 (+ 43us) client_negotiation.cc:240] Negotiation successful
0621 05:14:55.177113 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16}
I20260621 05:14:55.177539 26411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:14:55.169092 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41744 (local address 127.3.131.188:32823)
0621 05:14:55.169196 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0621 05:14:55.169197 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0621 05:14:55.169202 (+ 5us) server_negotiation.cc:407] Connection header received
0621 05:14:55.169213 (+ 11us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:14:55.169214 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:14:55.169221 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:14:55.169279 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:14:55.169562 (+ 283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:14:55.170119 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:14:55.171195 (+ 1076us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:14:55.171474 (+ 279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:14:55.171742 (+ 268us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:14:55.173831 (+ 2089us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:14:55.173832 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:14:55.173833 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:14:55.173840 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:14:55.175894 (+ 2054us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:14:55.176376 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:14:55.176377 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:14:55.176378 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:14:55.176436 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:14:55.176674 (+ 238us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:14:55.176675 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:14:55.176675 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:14:55.176855 (+ 180us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:14:55.177332 (+ 477us) server_negotiation.cc:1035] Waiting for connection context
0621 05:14:55.177376 (+ 44us) server_negotiation.cc:299] Negotiation successful
0621 05:14:55.177450 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":93}
I20260621 05:14:55.178275 26257 raft_consensus.cc:348] T 00000000000000000000000000000000 P 99b213827ec84131a97b66856d66ca8b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99b213827ec84131a97b66856d66ca8b" member_type: VOTER last_known_addr { host: "127.3.131.190" port: 36663 } } peers { permanent_uuid: "081e3ee07d0040bba3a6c9dea95c4698" member_type: VOTER last_known_addr { host: "127.3.131.189" port: 37625 } } peers { permanent_uuid: "6064b320c9fc476bb74da93a4021d44f" member_type: VOTER last_known_addr { host: "127.3.131.188" port: 32823 } }
I20260621 05:14:55.178431 26257 raft_consensus.cc:729] T 00000000000000000000000000000000 P 99b213827ec84131a97b66856d66ca8b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99b213827ec84131a97b66856d66ca8b, State: Initialized, Role: FOLLOWER
I20260621 05:14:55.178506 26257 consensus_queue.cc:260] T 00000000000000000000000000000000 P 99b213827ec84131a97b66856d66ca8b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99b213827ec84131a97b66856d66ca8b" member_type: VOTER last_known_addr { host: "127.3.131.190" port: 36663 } } peers { permanent_uuid: "081e3ee07d0040bba3a6c9dea95c4698" member_type: VOTER last_known_addr { host: "127.3.131.189" port: 37625 } } peers { permanent_uuid: "6064b320c9fc476bb74da93a4021d44f" member_type: VOTER last_known_addr { host: "127.3.131.188" port: 32823 } }
I20260621 05:14:55.178807 26414 sys_catalog.cc:455] T 00000000000000000000000000000000 P 99b213827ec84131a97b66856d66ca8b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99b213827ec84131a97b66856d66ca8b" member_type: VOTER last_known_addr { host: "127.3.131.190" port: 36663 } } peers { permanent_uuid: "081e3ee07d0040bba3a6c9dea95c4698" member_type: VOTER last_known_addr { host: "127.3.131.189" port: 37625 } } peers { permanent_uuid: "6064b320c9fc476bb74da93a4021d44f" member_type: VOTER last_known_addr { host: "127.3.131.188" port: 32823 } } }
I20260621 05:14:55.178893 26257 sys_catalog.cc:564] T 00000000000000000000000000000000 P 99b213827ec84131a97b66856d66ca8b [sys.catalog]: configured and running, proceeding with master startup.
I20260621 05:14:55.178989 26414 sys_catalog.cc:458] T 00000000000000000000000000000000 P 99b213827ec84131a97b66856d66ca8b [sys.catalog]: This master's current role is: FOLLOWER
I20260621 05:14:55.179128 26407 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6064b320c9fc476bb74da93a4021d44f: Bootstrap starting.
I20260621 05:14:55.180094 26407 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6064b320c9fc476bb74da93a4021d44f: Neither blocks nor log segments found. Creating new log.
I20260621 05:14:55.180464 26407 log.cc:824] T 00000000000000000000000000000000 P 6064b320c9fc476bb74da93a4021d44f: Log is configured to *not* fsync() on all Append() calls
W20260621 05:14:55.181440 26425 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 99b213827ec84131a97b66856d66ca8b: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260621 05:14:55.181505 26425 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260621 05:14:55.182237 26407 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6064b320c9fc476bb74da93a4021d44f: No bootstrap required, opened a new log
Jun 21 05:14:55 dist-test-slave-vrq1 krb5kdc[26176](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782018894, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.131.189@KRBTEST.COM
I20260621 05:14:55.184774 26407 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6064b320c9fc476bb74da93a4021d44f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99b213827ec84131a97b66856d66ca8b" member_type: VOTER last_known_addr { host: "127.3.131.190" port: 36663 } } peers { permanent_uuid: "081e3ee07d0040bba3a6c9dea95c4698" member_type: VOTER last_known_addr { host: "127.3.131.189" port: 37625 } } peers { permanent_uuid: "6064b320c9fc476bb74da93a4021d44f" member_type: VOTER last_known_addr { host: "127.3.131.188" port: 32823 } }
I20260621 05:14:55.184971 26407 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6064b320c9fc476bb74da93a4021d44f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6064b320c9fc476bb74da93a4021d44f, State: Initialized, Role: FOLLOWER
I20260621 05:14:55.185187 26407 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6064b320c9fc476bb74da93a4021d44f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99b213827ec84131a97b66856d66ca8b" member_type: VOTER last_known_addr { host: "127.3.131.190" port: 36663 } } peers { permanent_uuid: "081e3ee07d0040bba3a6c9dea95c4698" member_type: VOTER last_known_addr { host: "127.3.131.189" port: 37625 } } peers { permanent_uuid: "6064b320c9fc476bb74da93a4021d44f" member_type: VOTER last_known_addr { host: "127.3.131.188" port: 32823 } }
I20260621 05:14:55.185719 26407 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6064b320c9fc476bb74da93a4021d44f [sys.catalog]: configured and running, proceeding with master startup.
I20260621 05:14:55.182286 26189 master_runner.cc:186] Error getting master registration for 127.3.131.189:37625: OK, instance_id { permanent_uuid: "081e3ee07d0040bba3a6c9dea95c4698" instance_seqno: 1782018894972960 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260621 05:14:55.187009 26189 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260621 05:14:55.188158 26439 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 6064b320c9fc476bb74da93a4021d44f: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
I20260621 05:14:55.188308 26336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:14:55.177226 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54776 (local address 127.3.131.189:37625)
0621 05:14:55.177505 (+ 279us) server_negotiation.cc:206] Beginning negotiation
0621 05:14:55.177507 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:14:55.177514 (+ 7us) server_negotiation.cc:407] Connection header received
0621 05:14:55.177536 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:14:55.177537 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:14:55.177546 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:14:55.177622 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:14:55.178345 (+ 723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:14:55.179156 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:14:55.180143 (+ 987us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:14:55.180508 (+ 365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:14:55.180602 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:14:55.184187 (+ 3585us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:14:55.184188 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:14:55.184189 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:14:55.184199 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:14:55.186269 (+ 2070us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:14:55.186914 (+ 645us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:14:55.186916 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:14:55.186916 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:14:55.186985 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:14:55.187214 (+ 229us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:14:55.187215 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:14:55.187215 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:14:55.187474 (+ 259us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:14:55.187657 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0621 05:14:55.188130 (+ 473us) server_negotiation.cc:299] Negotiation successful
0621 05:14:55.188213 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":258}
W20260621 05:14:55.188948 26439 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260621 05:14:55.188687 26428 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6064b320c9fc476bb74da93a4021d44f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99b213827ec84131a97b66856d66ca8b" member_type: VOTER last_known_addr { host: "127.3.131.190" port: 36663 } } peers { permanent_uuid: "081e3ee07d0040bba3a6c9dea95c4698" member_type: VOTER last_known_addr { host: "127.3.131.189" port: 37625 } } peers { permanent_uuid: "6064b320c9fc476bb74da93a4021d44f" member_type: VOTER last_known_addr { host: "127.3.131.188" port: 32823 } } }
I20260621 05:14:55.189266 26428 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6064b320c9fc476bb74da93a4021d44f [sys.catalog]: This master's current role is: FOLLOWER
I20260621 05:14:55.189831 3598 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260621 05:14:55.190783 26231 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "99b213827ec84131a97b66856d66ca8b"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36932
I20260621 05:14:55.190922 26231 raft_consensus.cc:482] T 00000000000000000000000000000000 P 99b213827ec84131a97b66856d66ca8b [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260621 05:14:55.190986 26231 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 99b213827ec84131a97b66856d66ca8b [term 0 FOLLOWER]: Advancing to term 1
I20260621 05:14:55.189350 26339 master_runner.cc:186] Error getting master registration for 127.3.131.189:37625: OK, instance_id { permanent_uuid: "081e3ee07d0040bba3a6c9dea95c4698" instance_seqno: 1782018894972960 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260621 05:14:55.191797 26339 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260621 05:14:55.192433 26231 raft_consensus.cc:504] T 00000000000000000000000000000000 P 99b213827ec84131a97b66856d66ca8b [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99b213827ec84131a97b66856d66ca8b" member_type: VOTER last_known_addr { host: "127.3.131.190" port: 36663 } } peers { permanent_uuid: "081e3ee07d0040bba3a6c9dea95c4698" member_type: VOTER last_known_addr { host: "127.3.131.189" port: 37625 } } peers { permanent_uuid: "6064b320c9fc476bb74da93a4021d44f" member_type: VOTER last_known_addr { host: "127.3.131.188" port: 32823 } }
I20260621 05:14:55.192763 26231 leader_election.cc:288] T 00000000000000000000000000000000 P 99b213827ec84131a97b66856d66ca8b [CANDIDATE]: Term 1 election: Requested vote from peers 081e3ee07d0040bba3a6c9dea95c4698 (127.3.131.189:37625), 6064b320c9fc476bb74da93a4021d44f (127.3.131.188:32823)
I20260621 05:14:55.193028 26381 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "99b213827ec84131a97b66856d66ca8b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "6064b320c9fc476bb74da93a4021d44f"
I20260621 05:14:55.193162 26381 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6064b320c9fc476bb74da93a4021d44f [term 0 FOLLOWER]: Advancing to term 1
I20260621 05:14:55.194082 26381 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 6064b320c9fc476bb74da93a4021d44f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 99b213827ec84131a97b66856d66ca8b in term 1.
I20260621 05:14:55.194389 26207 leader_election.cc:302] T 00000000000000000000000000000000 P 99b213827ec84131a97b66856d66ca8b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6064b320c9fc476bb74da93a4021d44f, 99b213827ec84131a97b66856d66ca8b; no voters:
I20260621 05:14:55.194540 26414 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 99b213827ec84131a97b66856d66ca8b [term 1 FOLLOWER]: Leader election won for term 1
I20260621 05:14:55.194723 26414 raft_consensus.cc:686] T 00000000000000000000000000000000 P 99b213827ec84131a97b66856d66ca8b [term 1 LEADER]: Becoming Leader. State: Replica: 99b213827ec84131a97b66856d66ca8b, State: Running, Role: LEADER
I20260621 05:14:55.194844 26414 consensus_queue.cc:237] T 00000000000000000000000000000000 P 99b213827ec84131a97b66856d66ca8b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99b213827ec84131a97b66856d66ca8b" member_type: VOTER last_known_addr { host: "127.3.131.190" port: 36663 } } peers { permanent_uuid: "081e3ee07d0040bba3a6c9dea95c4698" member_type: VOTER last_known_addr { host: "127.3.131.189" port: 37625 } } peers { permanent_uuid: "6064b320c9fc476bb74da93a4021d44f" member_type: VOTER last_known_addr { host: "127.3.131.188" port: 32823 } }
I20260621 05:14:55.195464 26414 sys_catalog.cc:455] T 00000000000000000000000000000000 P 99b213827ec84131a97b66856d66ca8b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 99b213827ec84131a97b66856d66ca8b. Latest consensus state: current_term: 1 leader_uuid: "99b213827ec84131a97b66856d66ca8b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99b213827ec84131a97b66856d66ca8b" member_type: VOTER last_known_addr { host: "127.3.131.190" port: 36663 } } peers { permanent_uuid: "081e3ee07d0040bba3a6c9dea95c4698" member_type: VOTER last_known_addr { host: "127.3.131.189" port: 37625 } } peers { permanent_uuid: "6064b320c9fc476bb74da93a4021d44f" member_type: VOTER last_known_addr { host: "127.3.131.188" port: 32823 } } }
I20260621 05:14:55.195595 26414 sys_catalog.cc:458] T 00000000000000000000000000000000 P 99b213827ec84131a97b66856d66ca8b [sys.catalog]: This master's current role is: LEADER
I20260621 05:14:55.203763 26261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:14:55.194142 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36944 (local address 127.3.131.190:36663)
0621 05:14:55.194189 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0621 05:14:55.194191 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:14:55.194847 (+ 656us) server_negotiation.cc:407] Connection header received
0621 05:14:55.194949 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:14:55.194950 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:14:55.194961 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:14:55.194993 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:14:55.195852 (+ 859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:14:55.196368 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:14:55.197540 (+ 1172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:14:55.197753 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:14:55.197836 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:14:55.199731 (+ 1895us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:14:55.199732 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:14:55.199733 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:14:55.199741 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:14:55.202149 (+ 2408us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:14:55.202644 (+ 495us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:14:55.202645 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:14:55.202646 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:14:55.202732 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:14:55.202998 (+ 266us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:14:55.203000 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:14:55.203000 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:14:55.203166 (+ 166us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:14:55.203306 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0621 05:14:55.203535 (+ 229us) server_negotiation.cc:299] Negotiation successful
0621 05:14:55.203631 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260621 05:14:55.205451 26411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:14:55.194245 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41754 (local address 127.3.131.188:32823)
0621 05:14:55.194307 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0621 05:14:55.194310 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0621 05:14:55.194678 (+ 368us) server_negotiation.cc:407] Connection header received
0621 05:14:55.194747 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:14:55.194748 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:14:55.194760 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:14:55.194819 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:14:55.195785 (+ 966us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:14:55.196509 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:14:55.197912 (+ 1403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:14:55.198891 (+ 979us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:14:55.199011 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:14:55.200412 (+ 1401us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:14:55.200413 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:14:55.200414 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:14:55.200421 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:14:55.203729 (+ 3308us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:14:55.204586 (+ 857us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:14:55.204588 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:14:55.204589 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:14:55.204650 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:14:55.204859 (+ 209us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:14:55.204860 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:14:55.204861 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:14:55.205000 (+ 139us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:14:55.205102 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0621 05:14:55.205259 (+ 157us) server_negotiation.cc:299] Negotiation successful
0621 05:14:55.205352 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260621 05:14:55.210341 26332 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.131.188" port: 32823 } attempt: 3
I20260621 05:14:55.210630 26450 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260621 05:14:55.210901 26450 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260621 05:14:55.212464 26336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:14:55.194170 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54780 (local address 127.3.131.189:37625)
0621 05:14:55.194213 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0621 05:14:55.194215 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:14:55.203786 (+ 9571us) server_negotiation.cc:407] Connection header received
0621 05:14:55.203909 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:14:55.203910 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:14:55.203926 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:14:55.203963 (+ 37us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:14:55.204638 (+ 675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:14:55.205376 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:14:55.206489 (+ 1113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:14:55.206793 (+ 304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:14:55.206872 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:14:55.208423 (+ 1551us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:14:55.208424 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:14:55.208424 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:14:55.208432 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:14:55.210459 (+ 2027us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:14:55.211161 (+ 702us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:14:55.211162 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:14:55.211162 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:14:55.211280 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:14:55.211606 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:14:55.211607 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:14:55.211607 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:14:55.211751 (+ 144us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:14:55.211822 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0621 05:14:55.212256 (+ 434us) server_negotiation.cc:299] Negotiation successful
0621 05:14:55.212340 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260621 05:14:55.215557 26381 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 6064b320c9fc476bb74da93a4021d44f [term 1 FOLLOWER]: Refusing update from remote peer 99b213827ec84131a97b66856d66ca8b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
Jun 21 05:14:55 dist-test-slave-vrq1 krb5kdc[26176](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782018894, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.131.189@KRBTEST.COM for kudu/127.3.131.188@KRBTEST.COM
I20260621 05:14:55.216121 26448 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 99b213827ec84131a97b66856d66ca8b [LEADER]: Connected to new peer: Peer: permanent_uuid: "6064b320c9fc476bb74da93a4021d44f" member_type: VOTER last_known_addr { host: "127.3.131.188" port: 32823 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 05:14:55.218741 26428 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6064b320c9fc476bb74da93a4021d44f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 99b213827ec84131a97b66856d66ca8b. Latest consensus state: current_term: 1 leader_uuid: "99b213827ec84131a97b66856d66ca8b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99b213827ec84131a97b66856d66ca8b" member_type: VOTER last_known_addr { host: "127.3.131.190" port: 36663 } } peers { permanent_uuid: "081e3ee07d0040bba3a6c9dea95c4698" member_type: VOTER last_known_addr { host: "127.3.131.189" port: 37625 } } peers { permanent_uuid: "6064b320c9fc476bb74da93a4021d44f" member_type: VOTER last_known_addr { host: "127.3.131.188" port: 32823 } } }
I20260621 05:14:55.218852 26428 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6064b320c9fc476bb74da93a4021d44f [sys.catalog]: This master's current role is: FOLLOWER
I20260621 05:14:55.220834 26334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:14:55.210625 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.131.188:32823 (local address 127.0.0.1:41768)
0621 05:14:55.210721 (+ 96us) negotiation.cc:107] Waiting for socket to connect
0621 05:14:55.210738 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0621 05:14:55.210796 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:14:55.211153 (+ 357us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:14:55.211154 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:14:55.211354 (+ 200us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:14:55.211411 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:14:55.211413 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:14:55.212450 (+ 1037us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:14:55.212451 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:14:55.213229 (+ 778us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:14:55.213231 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:14:55.213861 (+ 630us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:14:55.213862 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:14:55.213923 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:14:55.214544 (+ 621us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 05:14:55.214562 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:14:55.216307 (+ 1745us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 05:14:55.218728 (+ 2421us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:14:55.218729 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:14:55.218729 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:14:55.218984 (+ 255us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:14:55.219663 (+ 679us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:14:55.219665 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:14:55.219665 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:14:55.219836 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:14:55.220348 (+ 512us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 05:14:55.220349 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 05:14:55.220441 (+ 92us) client_negotiation.cc:769] Sending connection context
0621 05:14:55.220570 (+ 129us) client_negotiation.cc:240] Negotiation successful
0621 05:14:55.220706 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":77}
I20260621 05:14:55.221357 26411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:14:55.210929 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41768 (local address 127.3.131.188:32823)
0621 05:14:55.211013 (+ 84us) server_negotiation.cc:206] Beginning negotiation
0621 05:14:55.211015 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:14:55.211033 (+ 18us) server_negotiation.cc:407] Connection header received
0621 05:14:55.211057 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:14:55.211058 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:14:55.211069 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:14:55.211481 (+ 412us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:14:55.211538 (+ 57us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:14:55.212361 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:14:55.213461 (+ 1100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:14:55.213768 (+ 307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:14:55.213853 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:14:55.216515 (+ 2662us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:14:55.216517 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:14:55.216517 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:14:55.216526 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:14:55.218599 (+ 2073us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:14:55.219491 (+ 892us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:14:55.219492 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:14:55.219492 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:14:55.219577 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:14:55.219934 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:14:55.219934 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:14:55.219935 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:14:55.220095 (+ 160us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:14:55.221043 (+ 948us) server_negotiation.cc:1035] Waiting for connection context
0621 05:14:55.221093 (+ 50us) server_negotiation.cc:299] Negotiation successful
0621 05:14:55.221184 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260621 05:14:55.222991 26332 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 081e3ee07d0040bba3a6c9dea95c4698: Bootstrap starting.
I20260621 05:14:55.223922 26332 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 081e3ee07d0040bba3a6c9dea95c4698: Neither blocks nor log segments found. Creating new log.
I20260621 05:14:55.224274 26332 log.cc:824] T 00000000000000000000000000000000 P 081e3ee07d0040bba3a6c9dea95c4698: Log is configured to *not* fsync() on all Append() calls
I20260621 05:14:55.225368 26332 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 081e3ee07d0040bba3a6c9dea95c4698: No bootstrap required, opened a new log
I20260621 05:14:55.228672 26332 raft_consensus.cc:348] T 00000000000000000000000000000000 P 081e3ee07d0040bba3a6c9dea95c4698 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99b213827ec84131a97b66856d66ca8b" member_type: VOTER last_known_addr { host: "127.3.131.190" port: 36663 } } peers { permanent_uuid: "081e3ee07d0040bba3a6c9dea95c4698" member_type: VOTER last_known_addr { host: "127.3.131.189" port: 37625 } } peers { permanent_uuid: "6064b320c9fc476bb74da93a4021d44f" member_type: VOTER last_known_addr { host: "127.3.131.188" port: 32823 } }
I20260621 05:14:55.228847 26332 raft_consensus.cc:729] T 00000000000000000000000000000000 P 081e3ee07d0040bba3a6c9dea95c4698 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 081e3ee07d0040bba3a6c9dea95c4698, State: Initialized, Role: FOLLOWER
I20260621 05:14:55.228942 26332 consensus_queue.cc:260] T 00000000000000000000000000000000 P 081e3ee07d0040bba3a6c9dea95c4698 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99b213827ec84131a97b66856d66ca8b" member_type: VOTER last_known_addr { host: "127.3.131.190" port: 36663 } } peers { permanent_uuid: "081e3ee07d0040bba3a6c9dea95c4698" member_type: VOTER last_known_addr { host: "127.3.131.189" port: 37625 } } peers { permanent_uuid: "6064b320c9fc476bb74da93a4021d44f" member_type: VOTER last_known_addr { host: "127.3.131.188" port: 32823 } }
I20260621 05:14:55.229362 26332 sys_catalog.cc:564] T 00000000000000000000000000000000 P 081e3ee07d0040bba3a6c9dea95c4698 [sys.catalog]: configured and running, proceeding with master startup.
I20260621 05:14:55.229422 26306 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "99b213827ec84131a97b66856d66ca8b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "081e3ee07d0040bba3a6c9dea95c4698"
I20260621 05:14:55.229519 26306 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 081e3ee07d0040bba3a6c9dea95c4698 [term 0 FOLLOWER]: Advancing to term 1
I20260621 05:14:55.230599 26306 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 081e3ee07d0040bba3a6c9dea95c4698 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 99b213827ec84131a97b66856d66ca8b in term 1.
I20260621 05:14:55.229292 26455 sys_catalog.cc:455] T 00000000000000000000000000000000 P 081e3ee07d0040bba3a6c9dea95c4698 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99b213827ec84131a97b66856d66ca8b" member_type: VOTER last_known_addr { host: "127.3.131.190" port: 36663 } } peers { permanent_uuid: "081e3ee07d0040bba3a6c9dea95c4698" member_type: VOTER last_known_addr { host: "127.3.131.189" port: 37625 } } peers { permanent_uuid: "6064b320c9fc476bb74da93a4021d44f" member_type: VOTER last_known_addr { host: "127.3.131.188" port: 32823 } } }
I20260621 05:14:55.230831 26305 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 081e3ee07d0040bba3a6c9dea95c4698 [term 1 FOLLOWER]: Refusing update from remote peer 99b213827ec84131a97b66856d66ca8b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 05:14:55.231366 26455 sys_catalog.cc:458] T 00000000000000000000000000000000 P 081e3ee07d0040bba3a6c9dea95c4698 [sys.catalog]: This master's current role is: FOLLOWER
I20260621 05:14:55.231798 26448 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 99b213827ec84131a97b66856d66ca8b [LEADER]: Connected to new peer: Peer: permanent_uuid: "081e3ee07d0040bba3a6c9dea95c4698" member_type: VOTER last_known_addr { host: "127.3.131.189" port: 37625 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260621 05:14:55.233093 26466 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 081e3ee07d0040bba3a6c9dea95c4698: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260621 05:14:55.233158 26466 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260621 05:14:55.235077 26448 sys_catalog.cc:455] T 00000000000000000000000000000000 P 99b213827ec84131a97b66856d66ca8b [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "99b213827ec84131a97b66856d66ca8b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99b213827ec84131a97b66856d66ca8b" member_type: VOTER last_known_addr { host: "127.3.131.190" port: 36663 } } peers { permanent_uuid: "081e3ee07d0040bba3a6c9dea95c4698" member_type: VOTER last_known_addr { host: "127.3.131.189" port: 37625 } } peers { permanent_uuid: "6064b320c9fc476bb74da93a4021d44f" member_type: VOTER last_known_addr { host: "127.3.131.188" port: 32823 } } }
I20260621 05:14:55.235267 26448 sys_catalog.cc:458] T 00000000000000000000000000000000 P 99b213827ec84131a97b66856d66ca8b [sys.catalog]: This master's current role is: LEADER
I20260621 05:14:55.235376 26451 mvcc.cc:205] Tried to move back new op lower bound from 7299149394799009792 to 7299149394718838784. Current Snapshot: MvccSnapshot[applied={T|T < 7299149394799009792}]
I20260621 05:14:55.236006 26428 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6064b320c9fc476bb74da93a4021d44f [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "99b213827ec84131a97b66856d66ca8b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99b213827ec84131a97b66856d66ca8b" member_type: VOTER last_known_addr { host: "127.3.131.190" port: 36663 } } peers { permanent_uuid: "081e3ee07d0040bba3a6c9dea95c4698" member_type: VOTER last_known_addr { host: "127.3.131.189" port: 37625 } } peers { permanent_uuid: "6064b320c9fc476bb74da93a4021d44f" member_type: VOTER last_known_addr { host: "127.3.131.188" port: 32823 } } }
I20260621 05:14:55.236102 26428 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6064b320c9fc476bb74da93a4021d44f [sys.catalog]: This master's current role is: FOLLOWER
I20260621 05:14:55.236444 26450 catalog_manager.cc:1347] Generated new cluster ID: 49ce7adbecda4ad58fb936908834a9d1
I20260621 05:14:55.236516 26450 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260621 05:14:55.237908 26452 mvcc.cc:205] Tried to move back new op lower bound from 7299149394799009792 to 7299149394718838784. Current Snapshot: MvccSnapshot[applied={T|T < 7299149394799009792}]
I20260621 05:14:55.238193 26455 sys_catalog.cc:455] T 00000000000000000000000000000000 P 081e3ee07d0040bba3a6c9dea95c4698 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 99b213827ec84131a97b66856d66ca8b. Latest consensus state: current_term: 1 leader_uuid: "99b213827ec84131a97b66856d66ca8b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99b213827ec84131a97b66856d66ca8b" member_type: VOTER last_known_addr { host: "127.3.131.190" port: 36663 } } peers { permanent_uuid: "081e3ee07d0040bba3a6c9dea95c4698" member_type: VOTER last_known_addr { host: "127.3.131.189" port: 37625 } } peers { permanent_uuid: "6064b320c9fc476bb74da93a4021d44f" member_type: VOTER last_known_addr { host: "127.3.131.188" port: 32823 } } }
I20260621 05:14:55.238291 26455 sys_catalog.cc:458] T 00000000000000000000000000000000 P 081e3ee07d0040bba3a6c9dea95c4698 [sys.catalog]: This master's current role is: FOLLOWER
I20260621 05:14:55.240528 26414 sys_catalog.cc:455] T 00000000000000000000000000000000 P 99b213827ec84131a97b66856d66ca8b [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "99b213827ec84131a97b66856d66ca8b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99b213827ec84131a97b66856d66ca8b" member_type: VOTER last_known_addr { host: "127.3.131.190" port: 36663 } } peers { permanent_uuid: "081e3ee07d0040bba3a6c9dea95c4698" member_type: VOTER last_known_addr { host: "127.3.131.189" port: 37625 } } peers { permanent_uuid: "6064b320c9fc476bb74da93a4021d44f" member_type: VOTER last_known_addr { host: "127.3.131.188" port: 32823 } } }
I20260621 05:14:55.240651 26414 sys_catalog.cc:458] T 00000000000000000000000000000000 P 99b213827ec84131a97b66856d66ca8b [sys.catalog]: This master's current role is: LEADER
I20260621 05:14:55.241149 26455 sys_catalog.cc:455] T 00000000000000000000000000000000 P 081e3ee07d0040bba3a6c9dea95c4698 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "99b213827ec84131a97b66856d66ca8b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99b213827ec84131a97b66856d66ca8b" member_type: VOTER last_known_addr { host: "127.3.131.190" port: 36663 } } peers { permanent_uuid: "081e3ee07d0040bba3a6c9dea95c4698" member_type: VOTER last_known_addr { host: "127.3.131.189" port: 37625 } } peers { permanent_uuid: "6064b320c9fc476bb74da93a4021d44f" member_type: VOTER last_known_addr { host: "127.3.131.188" port: 32823 } } }
I20260621 05:14:55.241252 26455 sys_catalog.cc:458] T 00000000000000000000000000000000 P 081e3ee07d0040bba3a6c9dea95c4698 [sys.catalog]: This master's current role is: FOLLOWER
I20260621 05:14:55.241514 26468 mvcc.cc:205] Tried to move back new op lower bound from 7299149394799009792 to 7299149394718838784. Current Snapshot: MvccSnapshot[applied={T|T < 7299149394799009792}]
I20260621 05:14:55.251552 26450 catalog_manager.cc:1370] Generated new certificate authority record
I20260621 05:14:55.252527 26450 catalog_manager.cc:1504] Loading token signing keys...
I20260621 05:14:55.264184 26450 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 99b213827ec84131a97b66856d66ca8b: Generated new TSK 0
I20260621 05:14:55.264456 26450 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260621 05:14:55.291476 26411 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 05:14:55.289701 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41770 (local address 127.3.131.188:32823)
0621 05:14:55.289747 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0621 05:14:55.289750 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0621 05:14:55.290727 (+ 977us) server_negotiation.cc:407] Connection header received
0621 05:14:55.290837 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:14:55.290838 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:14:55.290851 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:14:55.290883 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:14:55.291403 (+ 520us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41770: BlockingRecv error: recv got EOF from 127.0.0.1:41770 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
W20260621 05:14:55.292273 26336 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 05:14:55.290089 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54786 (local address 127.3.131.189:37625)
0621 05:14:55.290130 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0621 05:14:55.290131 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0621 05:14:55.291736 (+ 1605us) server_negotiation.cc:407] Connection header received
0621 05:14:55.291837 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:14:55.291838 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:14:55.291849 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:14:55.291879 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:14:55.292191 (+ 312us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54786: BlockingRecv error: recv got EOF from 127.0.0.1:54786 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
W20260621 05:14:55.292996 26261 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 05:14:55.290434 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36950 (local address 127.3.131.190:36663)
0621 05:14:55.290472 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0621 05:14:55.290474 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:14:55.292511 (+ 2037us) server_negotiation.cc:407] Connection header received
0621 05:14:55.292611 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:14:55.292612 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:14:55.292623 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:14:55.292652 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:14:55.292939 (+ 287us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36950: BlockingRecv error: recv got EOF from 127.0.0.1:36950 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
W20260621 05:14:55.295632 26336 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 05:14:55.294298 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54794 (local address 127.3.131.189:37625)
0621 05:14:55.294339 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0621 05:14:55.294341 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:14:55.294972 (+ 631us) server_negotiation.cc:407] Connection header received
0621 05:14:55.295110 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:14:55.295112 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:14:55.295134 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:14:55.295167 (+ 33us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:14:55.295570 (+ 403us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54794: BlockingRecv error: recv got EOF from 127.0.0.1:54794 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
W20260621 05:14:55.308241 26336 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 05:14:55.303648 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54810 (local address 127.3.131.189:37625)
0621 05:14:55.303697 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0621 05:14:55.303699 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:14:55.307563 (+ 3864us) server_negotiation.cc:407] Connection header received
0621 05:14:55.307686 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:14:55.307687 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:14:55.307701 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:14:55.307732 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:14:55.308174 (+ 442us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54810: BlockingRecv error: recv got EOF from 127.0.0.1:54810 (error 108)
Metrics: {"server-negotiator.queue_time_us":24}
W20260621 05:14:55.312659 26336 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 05:14:55.311648 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54826 (local address 127.3.131.189:37625)
0621 05:14:55.311693 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0621 05:14:55.311695 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:14:55.312014 (+ 319us) server_negotiation.cc:407] Connection header received
0621 05:14:55.312130 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:14:55.312131 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:14:55.312144 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:14:55.312176 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:14:55.312593 (+ 417us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54826: BlockingRecv error: recv got EOF from 127.0.0.1:54826 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
W20260621 05:14:55.319269 26336 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 05:14:55.318208 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54834 (local address 127.3.131.189:37625)
0621 05:14:55.318255 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0621 05:14:55.318257 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:14:55.318605 (+ 348us) server_negotiation.cc:407] Connection header received
0621 05:14:55.318717 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:14:55.318718 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:14:55.318731 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:14:55.318761 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:14:55.319169 (+ 408us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54834: BlockingRecv error: recv got EOF from 127.0.0.1:54834 (error 108)
Metrics: {"server-negotiator.queue_time_us":24}
W20260621 05:14:55.329696 26336 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 05:14:55.328708 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54836 (local address 127.3.131.189:37625)
0621 05:14:55.328756 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0621 05:14:55.328758 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:14:55.329070 (+ 312us) server_negotiation.cc:407] Connection header received
0621 05:14:55.329196 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:14:55.329197 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:14:55.329210 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:14:55.329241 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:14:55.329635 (+ 394us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54836: BlockingRecv error: recv got EOF from 127.0.0.1:54836 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
W20260621 05:14:55.348121 26336 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 05:14:55.347112 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54852 (local address 127.3.131.189:37625)
0621 05:14:55.347204 (+ 92us) server_negotiation.cc:206] Beginning negotiation
0621 05:14:55.347207 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0621 05:14:55.347504 (+ 297us) server_negotiation.cc:407] Connection header received
0621 05:14:55.347614 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:14:55.347615 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:14:55.347630 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:14:55.347662 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:14:55.348060 (+ 398us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54852: BlockingRecv error: recv got EOF from 127.0.0.1:54852 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
W20260621 05:14:55.382507 26336 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 05:14:55.381656 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54856 (local address 127.3.131.189:37625)
0621 05:14:55.381710 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0621 05:14:55.381712 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:14:55.381873 (+ 161us) server_negotiation.cc:407] Connection header received
0621 05:14:55.381955 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:14:55.381956 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:14:55.381971 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:14:55.382044 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:14:55.382420 (+ 376us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54856: BlockingRecv error: recv got EOF from 127.0.0.1:54856 (error 108)
Metrics: {"server-negotiator.queue_time_us":23}
W20260621 05:14:55.448812 26336 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 05:14:55.447907 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54866 (local address 127.3.131.189:37625)
0621 05:14:55.448011 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0621 05:14:55.448013 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:14:55.448138 (+ 125us) server_negotiation.cc:407] Connection header received
0621 05:14:55.448240 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:14:55.448241 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:14:55.448254 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:14:55.448334 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:14:55.448741 (+ 407us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54866: BlockingRecv error: recv got EOF from 127.0.0.1:54866 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
W20260621 05:14:55.579048 26336 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 05:14:55.578234 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54876 (local address 127.3.131.189:37625)
0621 05:14:55.578338 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0621 05:14:55.578340 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:14:55.578448 (+ 108us) server_negotiation.cc:407] Connection header received
0621 05:14:55.578539 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:14:55.578540 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:14:55.578552 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:14:55.578619 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:14:55.578996 (+ 377us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54876: BlockingRecv error: recv got EOF from 127.0.0.1:54876 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
W20260621 05:14:55.837325 26336 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 05:14:55.836510 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54880 (local address 127.3.131.189:37625)
0621 05:14:55.836604 (+ 94us) server_negotiation.cc:206] Beginning negotiation
0621 05:14:55.836606 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:14:55.836651 (+ 45us) server_negotiation.cc:407] Connection header received
0621 05:14:55.836681 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:14:55.836682 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:14:55.836692 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:14:55.836762 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:14:55.837255 (+ 493us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54880: BlockingRecv error: recv got EOF from 127.0.0.1:54880 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
I20260621 05:14:56.189651 26439 catalog_manager.cc:1259] Loaded cluster ID: 49ce7adbecda4ad58fb936908834a9d1
I20260621 05:14:56.189771 26439 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 6064b320c9fc476bb74da93a4021d44f: loading cluster ID for follower catalog manager: success
I20260621 05:14:56.190567 26439 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 6064b320c9fc476bb74da93a4021d44f: acquiring CA information for follower catalog manager: success
I20260621 05:14:56.190897 26439 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 6064b320c9fc476bb74da93a4021d44f: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260621 05:14:56.233632 26466 catalog_manager.cc:1259] Loaded cluster ID: 49ce7adbecda4ad58fb936908834a9d1
I20260621 05:14:56.233886 26466 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 081e3ee07d0040bba3a6c9dea95c4698: loading cluster ID for follower catalog manager: success
I20260621 05:14:56.234964 26466 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 081e3ee07d0040bba3a6c9dea95c4698: acquiring CA information for follower catalog manager: success
I20260621 05:14:56.235412 26466 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 081e3ee07d0040bba3a6c9dea95c4698: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260621 05:14:56.353328 26539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:14:56.350557 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54882 (local address 127.3.131.189:37625)
0621 05:14:56.350745 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0621 05:14:56.350747 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:14:56.350757 (+ 10us) server_negotiation.cc:407] Connection header received
0621 05:14:56.350787 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:14:56.350788 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:14:56.350849 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:14:56.350935 (+ 86us) server_negotiation.cc:226] Negotiated authn=TOKEN
0621 05:14:56.351092 (+ 157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:14:56.351678 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:14:56.352478 (+ 800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:14:56.352694 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:14:56.352773 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:14:56.352947 (+ 174us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0621 05:14:56.353072 (+ 125us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0621 05:14:56.353164 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0621 05:14:56.353207 (+ 43us) server_negotiation.cc:299] Negotiation successful
0621 05:14:56.353230 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":164,"thread_start_us":111,"threads_started":1}
I20260621 05:14:56.358127 26545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:14:56.355609 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41784 (local address 127.3.131.188:32823)
0621 05:14:56.355747 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0621 05:14:56.355749 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:14:56.355758 (+ 9us) server_negotiation.cc:407] Connection header received
0621 05:14:56.355804 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:14:56.355804 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:14:56.355847 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:14:56.355937 (+ 90us) server_negotiation.cc:226] Negotiated authn=TOKEN
0621 05:14:56.356046 (+ 109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:14:56.356567 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:14:56.357322 (+ 755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:14:56.357537 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:14:56.357618 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:14:56.357766 (+ 148us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0621 05:14:56.357885 (+ 119us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0621 05:14:56.357973 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0621 05:14:56.357996 (+ 23us) server_negotiation.cc:299] Negotiation successful
0621 05:14:56.358021 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":112,"thread_start_us":61,"threads_started":1}
I20260621 05:14:56.362870 26539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:14:56.359866 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54888 (local address 127.3.131.189:37625)
0621 05:14:56.359913 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0621 05:14:56.359915 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:14:56.360187 (+ 272us) server_negotiation.cc:407] Connection header received
0621 05:14:56.360256 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:14:56.360256 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:14:56.360269 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:14:56.360328 (+ 59us) server_negotiation.cc:226] Negotiated authn=TOKEN
0621 05:14:56.360425 (+ 97us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:14:56.361202 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:14:56.362026 (+ 824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:14:56.362305 (+ 279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:14:56.362366 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:14:56.362550 (+ 184us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0621 05:14:56.362591 (+ 41us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0621 05:14:56.362651 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0621 05:14:56.362730 (+ 79us) server_negotiation.cc:299] Negotiation successful
0621 05:14:56.362761 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260621 05:14:56.363240 26545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:14:56.359922 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41796 (local address 127.3.131.188:32823)
0621 05:14:56.359964 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0621 05:14:56.359966 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:14:56.360277 (+ 311us) server_negotiation.cc:407] Connection header received
0621 05:14:56.360368 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:14:56.360369 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:14:56.360379 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:14:56.360410 (+ 31us) server_negotiation.cc:226] Negotiated authn=TOKEN
0621 05:14:56.360595 (+ 185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:14:56.361068 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:14:56.362230 (+ 1162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:14:56.362508 (+ 278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:14:56.362565 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:14:56.362795 (+ 230us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0621 05:14:56.362835 (+ 40us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0621 05:14:56.362868 (+ 33us) server_negotiation.cc:1035] Waiting for connection context
0621 05:14:56.363119 (+ 251us) server_negotiation.cc:299] Negotiation successful
0621 05:14:56.363143 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260621 05:14:56.363447 26553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:14:56.360024 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36958 (local address 127.3.131.190:36663)
0621 05:14:56.360160 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0621 05:14:56.360162 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:14:56.360170 (+ 8us) server_negotiation.cc:407] Connection header received
0621 05:14:56.360221 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:14:56.360222 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:14:56.360281 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:14:56.360362 (+ 81us) server_negotiation.cc:226] Negotiated authn=TOKEN
0621 05:14:56.360553 (+ 191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:14:56.361302 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:14:56.362329 (+ 1027us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:14:56.362617 (+ 288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:14:56.362685 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:14:56.362942 (+ 257us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0621 05:14:56.363081 (+ 139us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0621 05:14:56.363138 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0621 05:14:56.363245 (+ 107us) server_negotiation.cc:299] Negotiation successful
0621 05:14:56.363271 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":113,"thread_start_us":85,"threads_started":1}
I20260621 05:14:56.365001 3598 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWZOx2_/build/release/bin/kudu with pid 26189
I20260621 05:14:56.370682 3598 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWZOx2_/build/release/bin/kudu with pid 26264
I20260621 05:14:56.375795 3598 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWZOx2_/build/release/bin/kudu with pid 26339
2026-06-21T05:14:56Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (2903 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (5882 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (279733 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST